jDiction Forum
English => Install & Configure jDiction => Topic started by: Finarfin on March 26, 2013, 08:15:09 pm
-
ok, easy question:
how do you configure jdiction to allow searchs in english in the english side and in german in the german side?
I saw a plugin for finder, that I suppose it is refering to the advanced searchs from joomla, but it seems I lack something.
Could you guide me? :)
-
The com_finder component has native multilang support and will automatically search the correct language.
But the jdiction finder content plugin is a extended version of the fonder content plugin to support translations done by jdiction it simulates multiple entries for one content entry. You need a own plugin for each component for jdiction translations.
Also you have to deactivate all other finder plugins.
-
Ok, lets see
Finder - jDiction Content, published
Smart search - Content, unpublished (from joomla)
Default language: spanish
Searching spanish word in spanish side: ok
Searching spanish word in english side: they appear, even when they musn´t
Searching english word in spanish side: nothing
Seraching english word in english side: nothing
-
Did you reset com_finder and have done a rescan?
-
ok, seems to work now :)
what about the normal search? its supported? :)
-
normal search is not supported, I think its useless ;-) maybe its possible but atm I can't touch this part.
-
Ok, I have this problem now:
Insert article and translate it.
Search for a term present on the translation.
The correct article will appear below BUT with the original title and content. The link appears translated. If you click on the link, it goes to the correct place nowadays.
Could you replicate it?
-
Actually, Im having more problems that I would like, dont know if the last changes have modified the finder behaviour >-<
-
Ok, I have this problem now:
Insert article and translate it.
Search for a term present on the translation.
The correct article will appear below BUT with the original title and content. The link appears translated. If you click on the link, it goes to the correct place nowadays.
Could you replicate it?
Partly the title is translated but the text is not. I will investigate...
While testing it seam that translations got deleted if Editor is used or maybe introtext and fulltext....
-
Hmm finder is working fine for me but there is a critical data loss problem when changing an item without touching the editor. (in com_content)
Fixed version attached.
I can't replicated your problem for com_finder, do you mean com_search?
-
I could try the new version to see if it works
About the finder, currently it seems that it doesn´t index all the entries. I don´t know why. When I use the index button, it stays on 99% and never finish. From command line it finish, but as I say, it lacks some entries that when you use the content finder from joomla works properly.
-
shouldn't happen its the same could with some modifications to load jdiction translation engine before load from database...
Checking with firebug could help you to find the problem, joomla makes multiple requests to get the status of indexing, maybe there is a problem in the replay.
-
Ok, I have to make more tests but there is a possibility that I found something.
There is any chance that the batch offset is being miscalculated? After some print_r in strategic places it seems like the subquery from content are something like
1000, 1001, 1002 ... 1099, 1100 and suddenly a gap where the ids start again at 1201 or something like that.
Could be that as each real content row is really two or more index entries (cause the several languages), it is miscalculated?
Maybe this is about the 99%. Could be that you don´t have this problem cause your batch size is less than your total article number...? mmm I´ll do some more test, let me know what you think.
-
The jd content finder plugin should be the as the original except that it return more row to the index.
So it could be possible that it confuse batch processing.
-
Do you have some idea of how to attack the problem?
-
Ok, I have news :D
Do you know when trying to find a bug you find like three more? :D well, there I am :)
About the finder, I have found three different possibilities that affect the results. I don´t know for sure how much affect each one until I make more tests but I´ll like to share what I found
- Another extension, virtual domains, could be interfering in the results because of the access levels.
- The offsett and batchsizes are calculated wrong when you do the Select * from content limit offset,batchsize, so you skip results
- And the most important now, on jdcontent, the return on line 370 is leaving out a lot of urls. For testing, I have simply commented it to see the results and they are starting to appear. I still have to understand the consequences of that, but Im on my way :D
So, that´s how I´m doing. I have to do more test and backtrack my changes, but I´m near, I feel it... :P
-
Commenting the line 370 provoques an memory limit overflow. Maybe I borked something on another place. A question related with this line, why did you search for ItemID? why not id?
-
I continue working on this. Now it seems that its not translating when inside de jdcontent.php. I mean, it generate several items for each entry, but they have the same content. And just as Im writing this I realize one possible problem. I almost always do the tests fom console, and that´s why is not translating. I mean, it probably don´t load properly jdiction (a quick test seems to confirm this)
I´m still suffering the bug where indexing trough the interface enters an infinite loop, and I have added a getCount to the adapter that is a patch (the original getCount won´t work properly, we execute each query several times so I´m multipliying for now)
If you could share some input on this I would be happy
-
Maybe you should reduce complexity, make a test installation with just so many articles you need.
Then use firebug to track the output of the indexing process.
-
New info
Almost 99% sure that (one) of the problems come with the getContentCount. The jdiction adapter needs a implementation of this method that takes into account the jdiction idiosincracy :P If I use the default one, I don´t get errors, but it lacks entries. If I use that number multiplied (by 2, by 3, as a guess of the real number), I get more and more until it breaks. So... the solution would be implementing a getContentCount that do something like "for each query, add the count to the total"
-
please add this function to lib/jdiction/finder/adapter.php
/**
* Method to get the number of content items available to index.
*
* @return integer The number of content items available to index.
*
* @since 2.5
* @throws Exception on database error.
*/
protected function getContentCount() {
$return = parent::getContentCount();
$jDiction = jDiction::getInstance();
$languages = $jDiction->getLanguages(true);
$return = $return * $languages;
return $return;
}
hopefully this fixes the problem.
-
It won´t :P If you return more items that it have, it breaks :D You have to return exactly or less. If more it will be a infinite loop (but you´ll be able to close it manually if you realize this)
-
Hmm it should not return more then it have.
getItems is doing the same, it takes the items and use each language for each row and return the result.
maybe it returns too many items because it doesn't really honor the $limit parameter.
You have to trace this because I don't have the time at the moment.
-
The fact is that the getContentCount have to be like this:
foreach language
total += getContentCount
and not
getContentCount * count(language)
What is the difference? That if I have a mix of translated and not translated articles, I will get different numbers for each getContentCount, depending of wich language I´m looking. Yes, the limit variable is working properly... except on the last iterations where I could get less.
english entry 1, translated to spanish
english entry 2, non translated
How many items I have to insert? not 4 (getcount * 2) but 3 (getcount with english + getcount with spanish)
I´ll let you know if I implement something that I can share :)
-
thats not correct
if your article is language * it will translated to english while indexing, one problem could be with articles that have a language set...
because jdiction adds this where statement:
$query->where('a.language in ('.$db->q('*').','.$db->q($language).')');
if we would only use
$query->where('a.language in ('.$db->q('*').')');
it should work, but misses articles in only one language.
-
Sorry, you are right, It will have problems only if you have articles with language different than "*"
-
Untested but tries this function:
protected function getContentCount() {
$jDiction = jDiction::getInstance();
$languages = $jDiction->getLanguages(true);
$return = 0;
// Get the list query.
$query = $this->getListQuery();
// Check if the query is valid.
if (empty($query)) {
return $return;
}
// we don't support non JDatabaseQueries here atm
if ($query instanceof JDatabaseQuery) {
return parent::getContentCount();
}
$columns = $this->db->getTableColumns($this->table);
if (array_key_exists('language', $columns)) {
$query = clone($query);
$query->clear('select')
->select('COUNT(*)')
->clear('order');
// Get the total number of content items per language to index.
foreach($languages as $language) {
$querylang = clone($query);
$querylang->where('a.language = '.$this->db->q($language->lang_code));
$this->db->setQuery($querylang);
$return += (int) $this->db->loadResult();
}
}
$querylang = clone($query);
// Get the total number of content items translated languages to index.
$querylang->where('a.language = '.$this->db->q('*'));
$this->db->setQuery($querylang);
$return += ((int) $this->db->loadResult() * count($languages));
return $return;
}
-
Almost perfect :D
Two points
Why the if ($query instanceof JDatabaseQuery) ? I don´t know why, but with that if it calls the parent and so give the bad number. I don´t know why is that the check, but I disable it for testing and worked. Perfectly! :O
Another topic, I guess that this
$querylang = clone($query);
// Get the total number of content items translated languages to index.
$querylang->where('a.language = '.$this->db->q('*'));
$this->db->setQuery($querylang);
$return += ((int) $this->db->loadResult() * count($languages));
have to be inside the if (array_key_exists('language', $columns)) {, but is a not really a big deal.
Another thing
Inside the getItems, you have to divide the offset between count(languages). Why? Because you are returning count(language)*number of items, but the table you are visiting is not queried as fast.
Example
Offset = 0
Limit = 60
getItems()
- select from 0 to 60 (for jdiction I will get 180)
Offset = 180
Limit = 60
here, if I do the select from 180 with a 60 limit, I´m leaving behind the entries between 60 and 180 so... I have to divide.
And that´s all for today. I want to clean my code, maybe do some tests more and I´ll be able to try to compilate the patches done so you have them available.
Regards and thanks for your help, that last function, saved me :P
-
You are right this function should work better:
protected function getContentCount() {
$jDiction = jDiction::getInstance();
$languages = $jDiction->getLanguages(true);
$return = 0;
// Get the list query.
$query = $this->getListQuery();
// Check if the query is valid.
if (empty($query)) {
return $return;
}
// we don't support non JDatabaseQueries here atm
if (!$query instanceof JDatabaseQuery) {
return parent::getContentCount();
}
$columns = $this->db->getTableColumns($this->table);
if (array_key_exists('language', $columns)) {
$query = clone($query);
$query->clear('select')
->select('COUNT(*)')
->clear('order');
// Get the total number of content items per language to index.
foreach($languages as $language) {
$querylang = clone($query);
$querylang->where('a.language = '.$this->db->q($language->lang_code));
$this->db->setQuery($querylang);
$return += (int) $this->db->loadResult();
}
// Get the total number of content items translated languages to index.
$query->where('a.language = '.$this->db->q('*'));
}
$this->db->setQuery($query);
$return += ((int) $this->db->loadResult() * count($languages));
return $return;
}
The Problem with the offset is that onBuildIndex calculates the batchOffset based on the rows it gets and not on the offset it gives me, jDictions returns more rows then expected...
I'm not sure how to solve this. Maybe by changing the onBuildIndex from:
// Adjust the offsets.
$offset++;
$iState->batchOffset++;
$iState->totalItems--;
to
// after the for loop
$offset += $offset;
$iState->batchOffset += $offset;
$iState->totalItems -= $offset;
but this would fail if we return less $items then offset... maybe changing this to count($items) if $offset is less then count($items).
-
Maybe this override could fix the count problem.
public function onBuildIndex()
{
// Get the indexer and adapter state.
$iState = FinderIndexer::getState();
$aState = $iState->pluginState[$this->context];
// Check the progress of the indexer and the adapter.
if ($iState->batchOffset == $iState->batchSize || $aState['offset'] == $aState['total'])
{
return true;
}
// Get the batch offset and size.
$offset = (int) $aState['offset'];
$limit = (int) ($iState->batchSize - $iState->batchOffset);
// Get the content items to index.
$items = $this->getItems($offset, $limit);
// Iterate through the items and index them.
for ($i = 0, $n = count($items); $i < $n; $i++)
{
// Index the item.
$this->index($items[$i]);
}
// Adjust the offsets.
if ($limit > count($items)) {
$limit = count($items);
}
$offset += $limit;
$iState->batchOffset += $limit;
$iState->totalItems -= $limit;
// Update the indexer state.
$aState['offset'] = $offset;
$iState->pluginState[$this->context] = $aState;
FinderIndexer::setState($iState);
return true;
}
-
Totalitems is not correctly calculated. You calculate totalitems before with the new getTotalCount.
I mean, getItems select 20 rows but return 40 items because of jdiction. Then, you increase the offset by 20 (ok), but reduce the totalItems by 20 (false, you´ll to do here totalitems -= count(items)
I´m going to keep yet the offset/count(languages) inside getItems. YOu know, is magic, dont touch it :D
In another topic, I don´t want to think too much if there would be problems if you have enough articles with languages differents than '*' :P
-
You are right, this should be better:
// Iterate through the items and index them.
for ($i = 0, $n = count($items); $i < $n; $i++)
{
// Index the item.
$this->index($items[$i]);
$iState->totalItems--;
}
// Adjust the offsets.
if ($limit > count($items)) {
$limit = count($items);
}
$offset += $limit;
$iState->batchOffset += $limit;
-
Well, I could clean the code so I´ll try to compilate:
- Bug, unresolved: Triying to execute the indexer from console will fail to load Jdiction, so it won´t translate properly the entries.
- jdcontent.php
- Line 287,
$params = $obj->params;
will generate notice or warnings, I have added a check here
if (isset($obj)) { $params = $obj->params }
- Line 366, there are two problematic lines
if (strpos($item->route, 'Itemid'))
but as long as I could recall, this comparison leave out a lot of entries. And this comparisons aren´t on the original content.php. I have changed them Itemid with id, but I guess that even this comparison is meaningless and unnecesary
- Line 450, function getListQuery
I have added this line before returning the query
$sql->order("a.id");
I don´t know why, but when I was doing the queries it returns me something like
id 21
id 22
id 23
id 18, this one entry is out of order, something that really doesn´t have to affect the results, but it seemed to do...
- adapter.php
- I´m going to keep my adapter version. I have being tracing my solution and is not efficient (it will reindex several times the same item), but it gets the job done. I have tried your onBuildIndex but doesn´t work, I think that is because of the way it calculates the limit. I would like to do more test and let you know but I´m near production and this is currently working :D:D
- Line 90, getItems function,
$db = JFactory::getDbo()
or
use $this->db
?
- Line 113, the solution I´m using
setQuery($query, $offset/count($languages), $limit)
instead of
setQuery($query, $offset, $limit)
the other posibility would be override onBuildIndex, but I don´t have the time to do the proper tests and this is working now :P
- Add the getContentCount function we said before
- jddatabase.php
- method_exists(parent, )
is wrong, the proper method to avoid notices is
method_exists(get_parent_class($this), )
There is at least two of this method_exists(parent). I could solve one, but the other one seems to break with the change.
- Unresolved bug, probably nothing related to jdiction: When the indexer tries to optimize a big table like the one generated, it breaks.
-
Well, I could clean the code so I´ll try to compilate:
- Bug, unresolved: Triying to execute the indexer from console will fail to load Jdiction, so it won´t translate properly the entries.
Didn't check this sorry
- jdcontent.php
- Line 287,
$params = $obj->params;
will generate notice or warnings, I have added a check here
if (isset($obj)) { $params = $obj->params }
Fixed
- Line 366, there are two problematic lines
if (strpos($item->route, 'Itemid'))
but as long as I could recall, this comparison leave out a lot of entries. And this comparisons aren´t on the original content.php. I have changed them Itemid with id, but I guess that even this comparison is meaningless and unnecesary
Fixed by removing this code, I filtered all articles out where no menu item and no category menu item exists, (used by me for special pages, but this is better done with meta data robots: "noindex" option
- Line 450, function getListQuery
I have added this line before returning the query
$sql->order("a.id");
I don´t know why, but when I was doing the queries it returns me something like
id 21
id 22
id 23
id 18, this one entry is out of order, something that really doesn´t have to affect the results, but it seemed to do...
thats hard to say why this is, maybe you print the query that get fired and try them in phpmyadmin
[li]adapter.php[/li]
- I´m going to keep my adapter version. I have being tracing my solution and is not efficient (it will reindex several times the same item), but it gets the job done. I have tried your onBuildIndex but doesn´t work, I think that is because of the way it calculates the limit. I would like to do more test and let you know but I´m near production and this is currently working :D:D
I followed your approach until I have time to fix it
- Line 90, getItems function,
$db = JFactory::getDbo()
or
use $this->db
?
Changed
- Line 113, the solution I´m using
setQuery($query, $offset/count($languages), $limit)
instead of
setQuery($query, $offset, $limit)
the other posibility would be override onBuildIndex, but I don´t have the time to do the proper tests and this is working now :P
As mentioned above I use the same code until I fix it.
- Add the getContentCount function we said before
done ;-)
[li]jddatabase.php[/li]
- method_exists(parent, )
is wrong, the proper method to avoid notices is
method_exists(get_parent_class($this), )
There is at least two of this method_exists(parent). I could solve one, but the other one seems to break with the change.
[/list]
I removed both calls to parent::__wakeup and parent::__sleep because they are not used in joomla atm.
- Unresolved bug, probably nothing related to jdiction: When the indexer tries to optimize a big table like the one generated, it breaks.
Did'nt tested this.
I attached the 3 files I changed so they should work with your version. I also updated some variables from joomla-master.