jDiction Forum

English => Install & Configure jDiction => Topic started by: Finarfin on March 26, 2013, 08:15:09 pm

Title: Searching with finder
Post by: Finarfin on March 26, 2013, 08:15:09 pm
ok, easy question:

how do you configure jdiction to allow searchs in english in the english side and in german in the german side?

I saw a plugin for finder, that I suppose it is refering to the advanced searchs from joomla, but it seems I lack something.

Could you guide me? :)
Title: Re: Searching with finder
Post by: Harald Leithner on March 26, 2013, 08:22:48 pm
The com_finder component has native multilang support and will automatically search the correct language.

But the jdiction finder content plugin is a extended version of the fonder content plugin to support translations done by jdiction it simulates multiple entries for one content entry. You need a own plugin for each component for jdiction translations.

Also you have to deactivate all other finder plugins.
Title: Re: Searching with finder
Post by: Finarfin on March 26, 2013, 09:25:24 pm
Ok, lets see

Finder - jDiction Content, published
Smart search - Content, unpublished (from joomla)

Default language: spanish

Searching spanish word in spanish side: ok
Searching spanish word in english side: they appear, even when they musn´t
Searching english word in spanish side: nothing
Seraching english word in english side: nothing

Title: Re: Searching with finder
Post by: Harald Leithner on March 26, 2013, 11:13:27 pm
Did you reset com_finder and have done a rescan?
Title: Re: Searching with finder
Post by: Finarfin on April 24, 2013, 03:23:24 pm
ok, seems to work now :)

what about the normal search? its supported? :)
Title: Re: Searching with finder
Post by: Harald Leithner on April 24, 2013, 03:26:38 pm
normal search is not supported, I think its useless ;-) maybe its possible but atm I can't touch this part.
Title: Re: Searching with finder
Post by: Finarfin on June 05, 2013, 09:38:43 pm
Ok, I have this problem now:

Insert article and translate it.
Search for a term present on the translation.
The correct article will appear below BUT with the original title and content. The link appears translated. If you click on the link, it goes to the correct place nowadays.

Could you replicate it?
Title: Re: Searching with finder
Post by: Finarfin on June 06, 2013, 01:11:20 am
Actually, Im having more problems that I would like, dont know if the last changes have modified the finder behaviour >-<
Title: Re: Searching with finder
Post by: Harald Leithner on June 06, 2013, 11:30:09 am
Ok, I have this problem now:

Insert article and translate it.
Search for a term present on the translation.
The correct article will appear below BUT with the original title and content. The link appears translated. If you click on the link, it goes to the correct place nowadays.

Could you replicate it?

Partly the title is translated but the text is not. I will investigate...

While testing it seam that translations got deleted if Editor is used or maybe introtext and fulltext....
Title: Re: Searching with finder
Post by: Harald Leithner on June 06, 2013, 02:36:25 pm
Hmm finder is working fine for me but there is a critical data loss problem when changing an item without touching the editor. (in com_content)

Fixed version attached.

I can't replicated your problem for com_finder, do you mean com_search?
Title: Re: Searching with finder
Post by: Finarfin on June 06, 2013, 04:02:34 pm
I could try the new version to see if it works



About the finder, currently it seems that it doesn´t index all the entries. I don´t know why. When I use the index button, it stays on 99% and never finish. From command line it finish, but as I say, it lacks some entries that when you use the content finder from joomla works properly.
Title: Re: Searching with finder
Post by: Harald Leithner on June 06, 2013, 06:18:18 pm
shouldn't happen its the same could with some modifications to load jdiction translation engine before load from database...

Checking with firebug could help you to find the problem, joomla makes multiple requests to get the status of indexing, maybe there is a problem in the replay.
Title: Re: Searching with finder
Post by: Finarfin on July 05, 2013, 08:17:27 pm
Ok, I have to make more tests but there is a possibility that I found something.

There is any chance that the batch offset is being miscalculated? After some print_r in strategic places it seems like the subquery from content are something like

1000, 1001, 1002 ... 1099, 1100 and suddenly a gap where the ids start again at 1201 or something like that.

Could be that as each real content row is really two or more index entries (cause the several languages), it is miscalculated?


Maybe this is about the 99%. Could be that you don´t have this problem cause your batch size is less than your total article number...? mmm I´ll do some more test, let me know what you think.
Title: Re: Searching with finder
Post by: Harald Leithner on July 05, 2013, 09:45:06 pm
The jd content finder plugin should be the as the original except that it return more row to the index.

So it could be possible that it confuse batch processing.
Title: Re: Searching with finder
Post by: Finarfin on July 07, 2013, 01:41:10 pm
Do you have some idea of how to attack the problem?
Title: Re: Searching with finder
Post by: Finarfin on July 09, 2013, 11:57:19 am
Ok, I have news :D

Do you know when trying to find a bug you find like three more? :D well, there I am :)

About the finder, I have found three different possibilities that affect the results. I don´t know for sure how much affect each one until I make more tests but I´ll like to share what I found

So, that´s how I´m doing. I have to do more test and backtrack my changes, but I´m near, I feel it... :P
Title: Re: Searching with finder
Post by: Finarfin on July 10, 2013, 07:58:13 pm
Commenting the line 370 provoques an memory limit overflow. Maybe I borked something on another place. A question related with this line, why did you search  for ItemID? why not id?
Title: Re: Searching with finder
Post by: Finarfin on July 11, 2013, 08:51:04 pm
I continue working on this. Now it seems that its not translating when inside de jdcontent.php. I mean, it generate several items for each entry, but they have the same content. And just as Im writing this I realize one possible problem. I almost always do the tests fom console, and that´s why is not translating. I mean, it probably don´t load properly jdiction (a quick test seems to confirm this)

I´m still suffering the bug where indexing trough the interface enters an infinite loop, and I have added a getCount to the adapter that is a patch (the original getCount won´t work properly, we execute each query several times so I´m multipliying for now)

If you could share some input on this I would be happy
Title: Re: Searching with finder
Post by: Harald Leithner on July 11, 2013, 09:02:50 pm
Maybe you should reduce complexity, make a test installation with just so many articles you need.

Then use firebug to track the output of the indexing process.
Title: Re: Searching with finder
Post by: Finarfin on July 12, 2013, 02:18:44 pm
New info

Almost 99% sure that (one) of the problems come with the getContentCount. The jdiction adapter needs a implementation of this method that takes into account the jdiction idiosincracy :P If I use the default one, I don´t get errors, but it lacks entries. If I use that number multiplied (by 2, by 3, as a guess of the real number), I get more and more until it breaks. So... the solution would be implementing a getContentCount that do something like "for each query, add the count to the total"

Title: Re: Searching with finder
Post by: Harald Leithner on July 12, 2013, 03:07:47 pm
please add this function to lib/jdiction/finder/adapter.php

Code: [Select]
  /**
   * Method to get the number of content items available to index.
   *
   * @return  integer  The number of content items available to index.
   *
   * @since   2.5
   * @throws  Exception on database error.
   */
  protected function getContentCount() {
    $return = parent::getContentCount();

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);
   
    $return = $return * $languages;
   
    return $return;
  }

hopefully this fixes the problem.
Title: Re: Searching with finder
Post by: Finarfin on July 12, 2013, 03:39:34 pm
It won´t :P If you return more items that it have, it breaks :D You have to return exactly or less. If more it will be a infinite loop (but you´ll be able to close it manually if you realize this)
Title: Re: Searching with finder
Post by: Harald Leithner on July 12, 2013, 03:45:02 pm
Hmm it should not return more then it have.

getItems is doing the same, it takes the items and use each language for each row and return the result.

maybe it returns too many items because it doesn't really honor the $limit parameter.

You have to trace this because I don't have the time at the moment.
Title: Re: Searching with finder
Post by: Finarfin on July 12, 2013, 04:29:45 pm
The fact is that the getContentCount have to be like this:


foreach language
total += getContentCount

and not

getContentCount * count(language)

What is the difference? That if I have a mix of translated and not translated articles, I will get different numbers for each getContentCount, depending of wich language I´m looking. Yes, the limit variable is working properly... except on the last iterations where I could get less.

english entry 1, translated to spanish
english entry 2, non translated

How many items I have to insert? not 4 (getcount * 2) but 3 (getcount with english + getcount with spanish)

I´ll let you know if I implement something that I can share :)
Title: Re: Searching with finder
Post by: Harald Leithner on July 12, 2013, 05:09:35 pm
thats not correct

if your article is language * it will translated to english while indexing, one problem could be with articles that have a language set...

because jdiction adds this where statement:

Code: [Select]
$query->where('a.language in ('.$db->q('*').','.$db->q($language).')');
if we would only use

Code: [Select]
$query->where('a.language in ('.$db->q('*').')');
it should work, but misses articles in only one language.
Title: Re: Searching with finder
Post by: Finarfin on July 12, 2013, 05:37:50 pm
Sorry, you are right, It will have problems only if you have articles with language different than "*"
Title: Re: Searching with finder
Post by: Harald Leithner on July 12, 2013, 06:21:05 pm
Untested but tries this function:

Code: [Select]
protected function getContentCount() {

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);

    $return = 0;

    // Get the list query.
    $query = $this->getListQuery();

    // Check if the query is valid.
    if (empty($query)) {
      return $return;
    }

    // we don't support non JDatabaseQueries here atm
    if ($query instanceof JDatabaseQuery) {
      return parent::getContentCount();
    }

    $columns = $this->db->getTableColumns($this->table);

    if (array_key_exists('language', $columns)) {
      $query = clone($query);
      $query->clear('select')
        ->select('COUNT(*)')
        ->clear('order');

      // Get the total number of content items per language to index.
      foreach($languages as $language) {

        $querylang = clone($query);
        $querylang->where('a.language = '.$this->db->q($language->lang_code));

        $this->db->setQuery($querylang);
        $return += (int) $this->db->loadResult();
      }
    }
    $querylang = clone($query);
    // Get the total number of content items translated languages to index.
    $querylang->where('a.language = '.$this->db->q('*'));
    $this->db->setQuery($querylang);
    $return += ((int) $this->db->loadResult() * count($languages));

    return $return;
  }
Title: Re: Searching with finder
Post by: Finarfin on July 12, 2013, 09:03:17 pm
Almost perfect :D

Two points


Why the if ($query instanceof JDatabaseQuery) ? I don´t know why, but with that if it calls the parent and so give the bad number. I don´t know why is that the check, but I disable it for testing and worked. Perfectly! :O

Another topic, I guess that this

    $querylang = clone($query);
    // Get the total number of content items translated languages to index.
    $querylang->where('a.language = '.$this->db->q('*'));
    $this->db->setQuery($querylang);
    $return += ((int) $this->db->loadResult() * count($languages));

have to be inside the if (array_key_exists('language', $columns)) {, but is a not really a big deal.




Another thing

Inside the getItems, you have to divide the offset between count(languages). Why? Because you are returning count(language)*number of items, but the table you are visiting is not queried as fast.

Example

Offset = 0
Limit = 60

getItems()
- select from 0 to 60 (for jdiction I will get 180)

Offset = 180
Limit = 60

here, if I do the select from 180 with a 60 limit, I´m leaving behind the entries between 60 and 180 so... I have to divide.

And that´s all for today. I want to clean my code, maybe do some tests more and I´ll be able to try to compilate the patches done so you have them available.

Regards and thanks for your help, that last function, saved me :P
Title: Re: Searching with finder
Post by: Harald Leithner on July 13, 2013, 12:08:30 am
You are right this function should work better:

Code: [Select]
protected function getContentCount() {

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);

    $return = 0;

    // Get the list query.
    $query = $this->getListQuery();

    // Check if the query is valid.
    if (empty($query)) {
      return $return;
    }

    // we don't support non JDatabaseQueries here atm
    if (!$query instanceof JDatabaseQuery) {
      return parent::getContentCount();
    }

    $columns = $this->db->getTableColumns($this->table);

    if (array_key_exists('language', $columns)) {
      $query = clone($query);
      $query->clear('select')
        ->select('COUNT(*)')
        ->clear('order');

      // Get the total number of content items per language to index.
      foreach($languages as $language) {

        $querylang = clone($query);
        $querylang->where('a.language = '.$this->db->q($language->lang_code));

        $this->db->setQuery($querylang);
        $return += (int) $this->db->loadResult();
      }

      // Get the total number of content items translated languages to index.
      $query->where('a.language = '.$this->db->q('*'));
    }
    $this->db->setQuery($query);
    $return += ((int) $this->db->loadResult() * count($languages));

    return $return;
  }

The Problem with the offset is that onBuildIndex calculates the batchOffset based on the rows it gets and not on the offset it gives me, jDictions returns more rows then expected...

I'm not sure how to solve this. Maybe by changing the onBuildIndex from:
Code: [Select]
// Adjust the offsets.
$offset++;
$iState->batchOffset++;
$iState->totalItems--;

to

Code: [Select]

// after the for loop
$offset += $offset;
$iState->batchOffset += $offset;
$iState->totalItems -= $offset;

but this would fail if we return less $items then offset... maybe changing this to count($items) if $offset is less then count($items).
Title: Re: Searching with finder
Post by: Harald Leithner on July 13, 2013, 12:53:38 pm
Maybe this override could fix the count problem.

Code: [Select]
  public function onBuildIndex()
  {
    // Get the indexer and adapter state.
    $iState = FinderIndexer::getState();
    $aState = $iState->pluginState[$this->context];

    // Check the progress of the indexer and the adapter.
    if ($iState->batchOffset == $iState->batchSize || $aState['offset'] == $aState['total'])
    {
      return true;
    }

    // Get the batch offset and size.
    $offset = (int) $aState['offset'];
    $limit = (int) ($iState->batchSize - $iState->batchOffset);

    // Get the content items to index.
    $items = $this->getItems($offset, $limit);

    // Iterate through the items and index them.
    for ($i = 0, $n = count($items); $i < $n; $i++)
    {
      // Index the item.
      $this->index($items[$i]);

    }

    // Adjust the offsets.
    if ($limit > count($items)) {
      $limit = count($items);
    }
    $offset += $limit;
    $iState->batchOffset += $limit;
    $iState->totalItems -= $limit;

    // Update the indexer state.
    $aState['offset'] = $offset;
    $iState->pluginState[$this->context] = $aState;
    FinderIndexer::setState($iState);

    return true;
  }
Title: Re: Searching with finder
Post by: Finarfin on July 13, 2013, 03:18:04 pm
Totalitems is not correctly calculated. You calculate totalitems before with the new getTotalCount.

I mean, getItems select 20 rows but return 40 items because of jdiction. Then, you increase the offset by 20 (ok), but reduce the totalItems by 20 (false, you´ll to do here totalitems -= count(items)

I´m going to keep yet the offset/count(languages) inside getItems. YOu know, is magic, dont touch it :D



In another topic, I don´t want to think too much if there would be problems if you have enough articles with languages differents than '*' :P
Title: Re: Searching with finder
Post by: Harald Leithner on July 13, 2013, 05:41:17 pm
You are right, this should be better:

Code: [Select]
// Iterate through the items and index them.
    for ($i = 0, $n = count($items); $i < $n; $i++)
    {
      // Index the item.
      $this->index($items[$i]);

      $iState->totalItems--;
    }

    // Adjust the offsets.
    if ($limit > count($items)) {
      $limit = count($items);
    }
    $offset += $limit;
    $iState->batchOffset += $limit;
Title: Re: Searching with finder
Post by: Finarfin on July 13, 2013, 09:32:45 pm
Well, I could clean the code so I´ll try to compilate:



Title: Re: Searching with finder
Post by: Harald Leithner on July 14, 2013, 12:57:30 pm
Well, I could clean the code so I´ll try to compilate:


  • Bug, unresolved: Triying to execute the indexer from console will fail to load Jdiction, so it won´t translate properly the entries.
Didn't check this sorry
  • jdcontent.php
    • Line 287,

      $params = $obj->params;

      will generate notice or warnings, I have added a check here

      if (isset($obj)) { $params = $obj->params }
Fixed
  • Line 366, there are two problematic lines

    if (strpos($item->route, 'Itemid'))

    but as long as I could recall, this comparison leave out a lot of entries. And this comparisons aren´t on the original content.php. I have changed them Itemid with id, but I guess that even this comparison is meaningless and unnecesary
Fixed by removing this code, I filtered all articles out where no menu item and no category menu item exists, (used by me for special pages, but this is better done with meta data robots: "noindex" option
  • Line 450, function getListQuery

    I have added this line before returning the query

    $sql->order("a.id");

    I don´t know why, but when I was doing the queries it returns me something like

    id 21
    id 22
    id 23
    id 18, this one entry is out of order, something that really doesn´t have to affect the results, but it seemed to do...
thats hard to say why this is, maybe you print the query that get fired and try them in phpmyadmin
    [li]adapter.php[/li]
    • I´m going to keep my adapter version. I have being tracing my solution and is not efficient (it will reindex several times the same item), but it gets the job done. I have tried your onBuildIndex but doesn´t work, I think that is because of the way it calculates the limit. I would like to do more test and let you know but I´m near production and this is currently working :D:D
    I followed your approach until I have time to fix it
    • Line 90, getItems function,

      $db = JFactory::getDbo()

      or

      use $this->db
      ?
    Changed
    • Line 113, the solution I´m using

      setQuery($query, $offset/count($languages), $limit)

      instead of
      setQuery($query, $offset, $limit)

      the other posibility would be override onBuildIndex, but I don´t have the time to do the proper tests and this is working now :P
    As mentioned above I use the same code until I fix it.
    • Add the getContentCount function we said before
    done ;-)
      [li]jddatabase.php[/li]
      • method_exists(parent, )
        is wrong, the proper method to avoid notices is
        method_exists(get_parent_class($this), )

        There is at least two of this method_exists(parent). I could solve one, but the other one seems to break with the change.
      [/list]
      I removed both calls to parent::__wakeup and parent::__sleep because they are not used in joomla atm.
      • Unresolved bug, probably nothing related to jdiction: When the indexer tries to optimize a big table like the one generated, it breaks.
      Did'nt tested this.


      I attached the 3 files I changed so they should work with your version. I also updated some variables from joomla-master.