jDiction Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

jDiction 2.2.0 released. http://jdiction.org/downloads

Pages: 1 [2] 3

Author Topic: Searching with finder  (Read 33427 times)

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #15 on: July 09, 2013, 11:57:19 am »

Ok, I have news :D

Do you know when trying to find a bug you find like three more? :D well, there I am :)

About the finder, I have found three different possibilities that affect the results. I don´t know for sure how much affect each one until I make more tests but I´ll like to share what I found
  • Another extension, virtual domains, could be interfering in the results because of the access levels.
  • The offsett and batchsizes are calculated wrong when you do the Select * from content limit offset,batchsize, so you skip results
  • And the most important now, on jdcontent, the return on line 370 is leaving out a lot of urls. For testing, I have simply commented it to see the results and they are starting to appear. I still have to understand the consequences of that, but Im on my way :D

So, that´s how I´m doing. I have to do more test and backtrack my changes, but I´m near, I feel it... :P
Logged

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #16 on: July 10, 2013, 07:58:13 pm »

Commenting the line 370 provoques an memory limit overflow. Maybe I borked something on another place. A question related with this line, why did you search  for ItemID? why not id?
Logged

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #17 on: July 11, 2013, 08:51:04 pm »

I continue working on this. Now it seems that its not translating when inside de jdcontent.php. I mean, it generate several items for each entry, but they have the same content. And just as Im writing this I realize one possible problem. I almost always do the tests fom console, and that´s why is not translating. I mean, it probably don´t load properly jdiction (a quick test seems to confirm this)

I´m still suffering the bug where indexing trough the interface enters an infinite loop, and I have added a getCount to the adapter that is a patch (the original getCount won´t work properly, we execute each query several times so I´m multipliying for now)

If you could share some input on this I would be happy
Logged

Harald Leithner

  • Administrator
  • Hero Member
  • *****
  • Posts: 1684
    • View Profile
Re: Searching with finder
« Reply #18 on: July 11, 2013, 09:02:50 pm »

Maybe you should reduce complexity, make a test installation with just so many articles you need.

Then use firebug to track the output of the indexing process.
Logged
Joomla! 5.0 Release Manager
Vote at JED

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #19 on: July 12, 2013, 02:18:44 pm »

New info

Almost 99% sure that (one) of the problems come with the getContentCount. The jdiction adapter needs a implementation of this method that takes into account the jdiction idiosincracy :P If I use the default one, I don´t get errors, but it lacks entries. If I use that number multiplied (by 2, by 3, as a guess of the real number), I get more and more until it breaks. So... the solution would be implementing a getContentCount that do something like "for each query, add the count to the total"

Logged

Harald Leithner

  • Administrator
  • Hero Member
  • *****
  • Posts: 1684
    • View Profile
Re: Searching with finder
« Reply #20 on: July 12, 2013, 03:07:47 pm »

please add this function to lib/jdiction/finder/adapter.php

Code: [Select]
  /**
   * Method to get the number of content items available to index.
   *
   * @return  integer  The number of content items available to index.
   *
   * @since   2.5
   * @throws  Exception on database error.
   */
  protected function getContentCount() {
    $return = parent::getContentCount();

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);
   
    $return = $return * $languages;
   
    return $return;
  }

hopefully this fixes the problem.
Logged
Joomla! 5.0 Release Manager
Vote at JED

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #21 on: July 12, 2013, 03:39:34 pm »

It won´t :P If you return more items that it have, it breaks :D You have to return exactly or less. If more it will be a infinite loop (but you´ll be able to close it manually if you realize this)
Logged

Harald Leithner

  • Administrator
  • Hero Member
  • *****
  • Posts: 1684
    • View Profile
Re: Searching with finder
« Reply #22 on: July 12, 2013, 03:45:02 pm »

Hmm it should not return more then it have.

getItems is doing the same, it takes the items and use each language for each row and return the result.

maybe it returns too many items because it doesn't really honor the $limit parameter.

You have to trace this because I don't have the time at the moment.
Logged
Joomla! 5.0 Release Manager
Vote at JED

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #23 on: July 12, 2013, 04:29:45 pm »

The fact is that the getContentCount have to be like this:


foreach language
total += getContentCount

and not

getContentCount * count(language)

What is the difference? That if I have a mix of translated and not translated articles, I will get different numbers for each getContentCount, depending of wich language I´m looking. Yes, the limit variable is working properly... except on the last iterations where I could get less.

english entry 1, translated to spanish
english entry 2, non translated

How many items I have to insert? not 4 (getcount * 2) but 3 (getcount with english + getcount with spanish)

I´ll let you know if I implement something that I can share :)
Logged

Harald Leithner

  • Administrator
  • Hero Member
  • *****
  • Posts: 1684
    • View Profile
Re: Searching with finder
« Reply #24 on: July 12, 2013, 05:09:35 pm »

thats not correct

if your article is language * it will translated to english while indexing, one problem could be with articles that have a language set...

because jdiction adds this where statement:

Code: [Select]
$query->where('a.language in ('.$db->q('*').','.$db->q($language).')');
if we would only use

Code: [Select]
$query->where('a.language in ('.$db->q('*').')');
it should work, but misses articles in only one language.
Logged
Joomla! 5.0 Release Manager
Vote at JED

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #25 on: July 12, 2013, 05:37:50 pm »

Sorry, you are right, It will have problems only if you have articles with language different than "*"
Logged

Harald Leithner

  • Administrator
  • Hero Member
  • *****
  • Posts: 1684
    • View Profile
Re: Searching with finder
« Reply #26 on: July 12, 2013, 06:21:05 pm »

Untested but tries this function:

Code: [Select]
protected function getContentCount() {

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);

    $return = 0;

    // Get the list query.
    $query = $this->getListQuery();

    // Check if the query is valid.
    if (empty($query)) {
      return $return;
    }

    // we don't support non JDatabaseQueries here atm
    if ($query instanceof JDatabaseQuery) {
      return parent::getContentCount();
    }

    $columns = $this->db->getTableColumns($this->table);

    if (array_key_exists('language', $columns)) {
      $query = clone($query);
      $query->clear('select')
        ->select('COUNT(*)')
        ->clear('order');

      // Get the total number of content items per language to index.
      foreach($languages as $language) {

        $querylang = clone($query);
        $querylang->where('a.language = '.$this->db->q($language->lang_code));

        $this->db->setQuery($querylang);
        $return += (int) $this->db->loadResult();
      }
    }
    $querylang = clone($query);
    // Get the total number of content items translated languages to index.
    $querylang->where('a.language = '.$this->db->q('*'));
    $this->db->setQuery($querylang);
    $return += ((int) $this->db->loadResult() * count($languages));

    return $return;
  }
Logged
Joomla! 5.0 Release Manager
Vote at JED

Finarfin

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Searching with finder
« Reply #27 on: July 12, 2013, 09:03:17 pm »

Almost perfect :D

Two points


Why the if ($query instanceof JDatabaseQuery) ? I don´t know why, but with that if it calls the parent and so give the bad number. I don´t know why is that the check, but I disable it for testing and worked. Perfectly! :O

Another topic, I guess that this

    $querylang = clone($query);
    // Get the total number of content items translated languages to index.
    $querylang->where('a.language = '.$this->db->q('*'));
    $this->db->setQuery($querylang);
    $return += ((int) $this->db->loadResult() * count($languages));

have to be inside the if (array_key_exists('language', $columns)) {, but is a not really a big deal.




Another thing

Inside the getItems, you have to divide the offset between count(languages). Why? Because you are returning count(language)*number of items, but the table you are visiting is not queried as fast.

Example

Offset = 0
Limit = 60

getItems()
- select from 0 to 60 (for jdiction I will get 180)

Offset = 180
Limit = 60

here, if I do the select from 180 with a 60 limit, I´m leaving behind the entries between 60 and 180 so... I have to divide.

And that´s all for today. I want to clean my code, maybe do some tests more and I´ll be able to try to compilate the patches done so you have them available.

Regards and thanks for your help, that last function, saved me :P
Logged

Harald Leithner

  • Administrator
  • Hero Member
  • *****
  • Posts: 1684
    • View Profile
Re: Searching with finder
« Reply #28 on: July 13, 2013, 12:08:30 am »

You are right this function should work better:

Code: [Select]
protected function getContentCount() {

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);

    $return = 0;

    // Get the list query.
    $query = $this->getListQuery();

    // Check if the query is valid.
    if (empty($query)) {
      return $return;
    }

    // we don't support non JDatabaseQueries here atm
    if (!$query instanceof JDatabaseQuery) {
      return parent::getContentCount();
    }

    $columns = $this->db->getTableColumns($this->table);

    if (array_key_exists('language', $columns)) {
      $query = clone($query);
      $query->clear('select')
        ->select('COUNT(*)')
        ->clear('order');

      // Get the total number of content items per language to index.
      foreach($languages as $language) {

        $querylang = clone($query);
        $querylang->where('a.language = '.$this->db->q($language->lang_code));

        $this->db->setQuery($querylang);
        $return += (int) $this->db->loadResult();
      }

      // Get the total number of content items translated languages to index.
      $query->where('a.language = '.$this->db->q('*'));
    }
    $this->db->setQuery($query);
    $return += ((int) $this->db->loadResult() * count($languages));

    return $return;
  }

The Problem with the offset is that onBuildIndex calculates the batchOffset based on the rows it gets and not on the offset it gives me, jDictions returns more rows then expected...

I'm not sure how to solve this. Maybe by changing the onBuildIndex from:
Code: [Select]
// Adjust the offsets.
$offset++;
$iState->batchOffset++;
$iState->totalItems--;

to

Code: [Select]

// after the for loop
$offset += $offset;
$iState->batchOffset += $offset;
$iState->totalItems -= $offset;

but this would fail if we return less $items then offset... maybe changing this to count($items) if $offset is less then count($items).
« Last Edit: July 13, 2013, 12:48:11 pm by Harald Leithner »
Logged
Joomla! 5.0 Release Manager
Vote at JED

Harald Leithner

  • Administrator
  • Hero Member
  • *****
  • Posts: 1684
    • View Profile
Re: Searching with finder
« Reply #29 on: July 13, 2013, 12:53:38 pm »

Maybe this override could fix the count problem.

Code: [Select]
  public function onBuildIndex()
  {
    // Get the indexer and adapter state.
    $iState = FinderIndexer::getState();
    $aState = $iState->pluginState[$this->context];

    // Check the progress of the indexer and the adapter.
    if ($iState->batchOffset == $iState->batchSize || $aState['offset'] == $aState['total'])
    {
      return true;
    }

    // Get the batch offset and size.
    $offset = (int) $aState['offset'];
    $limit = (int) ($iState->batchSize - $iState->batchOffset);

    // Get the content items to index.
    $items = $this->getItems($offset, $limit);

    // Iterate through the items and index them.
    for ($i = 0, $n = count($items); $i < $n; $i++)
    {
      // Index the item.
      $this->index($items[$i]);

    }

    // Adjust the offsets.
    if ($limit > count($items)) {
      $limit = count($items);
    }
    $offset += $limit;
    $iState->batchOffset += $limit;
    $iState->totalItems -= $limit;

    // Update the indexer state.
    $aState['offset'] = $offset;
    $iState->pluginState[$this->context] = $aState;
    FinderIndexer::setState($iState);

    return true;
  }
Logged
Joomla! 5.0 Release Manager
Vote at JED

Pages: 1 [2] 3