Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Finarfin]

Ok, I have news

Do you know when trying to find a bug you find like three more? well, there I am

About the finder, I have found three different possibilities that affect the results. I don´t know for sure how much affect each one until I make more tests but I´ll like to share what I found

• Another extension, virtual domains, could be interfering in the results because of the access levels.
• The offsett and batchsizes are calculated wrong when you do the Select * from content limit offset,batchsize, so you skip results
• And the most important now, on jdcontent, the return on line 370 is leaving out a lot of urls. For testing, I have simply commented it to see the results and they are starting to appear. I still have to understand the consequences of that, but Im on my way

So, that´s how I´m doing. I have to do more test and backtrack my changes, but I´m near, I feel it...

[Finarfin]

Commenting the line 370 provoques an memory limit overflow. Maybe I borked something on another place. A question related with this line, why did you search  for ItemID? why not id?

[Finarfin]

I continue working on this. Now it seems that its not translating when inside de jdcontent.php. I mean, it generate several items for each entry, but they have the same content. And just as Im writing this I realize one possible problem. I almost always do the tests fom console, and that´s why is not translating. I mean, it probably don´t load properly jdiction (a quick test seems to confirm this)

I´m still suffering the bug where indexing trough the interface enters an infinite loop, and I have added a getCount to the adapter that is a patch (the original getCount won´t work properly, we execute each query several times so I´m multipliying for now)

If you could share some input on this I would be happy

[Harald Leithner]

Maybe you should reduce complexity, make a test installation with just so many articles you need.

Then use firebug to track the output of the indexing process.

[Finarfin]

New info

Almost 99% sure that (one) of the problems come with the getContentCount. The jdiction adapter needs a implementation of this method that takes into account the jdiction idiosincracy If I use the default one, I don´t get errors, but it lacks entries. If I use that number multiplied (by 2, by 3, as a guess of the real number), I get more and more until it breaks. So... the solution would be implementing a getContentCount that do something like "for each query, add the count to the total"

[Harald Leithner]

please add this function to lib/jdiction/finder/adapter.php

Code: [Select]

  /**
   * Method to get the number of content items available to index.
   *
   * @return  integer  The number of content items available to index.
   *
   * @since   2.5
   * @throws  Exception on database error.
   */
  protected function getContentCount() {
    $return = parent::getContentCount();

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);
   
    $return = $return * $languages;
   
    return $return;
  }
hopefully this fixes the problem.

[Finarfin]

It won´t If you return more items that it have, it breaks You have to return exactly or less. If more it will be a infinite loop (but you´ll be able to close it manually if you realize this)

[Harald Leithner]

Hmm it should not return more then it have.

getItems is doing the same, it takes the items and use each language for each row and return the result.

maybe it returns too many items because it doesn't really honor the $limit parameter.

You have to trace this because I don't have the time at the moment.

[Finarfin]

The fact is that the getContentCount have to be like this:


foreach language
total += getContentCount

and not

getContentCount * count(language)

What is the difference? That if I have a mix of translated and not translated articles, I will get different numbers for each getContentCount, depending of wich language I´m looking. Yes, the limit variable is working properly... except on the last iterations where I could get less.

english entry 1, translated to spanish
english entry 2, non translated

How many items I have to insert? not 4 (getcount * 2) but 3 (getcount with english + getcount with spanish)

I´ll let you know if I implement something that I can share

[Harald Leithner]

thats not correct

if your article is language * it will translated to english while indexing, one problem could be with articles that have a language set...

because jdiction adds this where statement:

Code: [Select]

$query->where('a.language in ('.$db->q('*').','.$db->q($language).')');
if we would only use

Code: [Select]

$query->where('a.language in ('.$db->q('*').')');
it should work, but misses articles in only one language.

[Finarfin]

Sorry, you are right, It will have problems only if you have articles with language different than "*"

[Harald Leithner]

Untested but tries this function:

Code: [Select]

protected function getContentCount() {

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);

    $return = 0;

    // Get the list query.
    $query = $this->getListQuery();

    // Check if the query is valid.
    if (empty($query)) {
      return $return;
    }

    // we don't support non JDatabaseQueries here atm
    if ($query instanceof JDatabaseQuery) {
      return parent::getContentCount();
    }

    $columns = $this->db->getTableColumns($this->table);

    if (array_key_exists('language', $columns)) {
      $query = clone($query);
      $query->clear('select')
        ->select('COUNT(*)')
        ->clear('order');

      // Get the total number of content items per language to index.
      foreach($languages as $language) {

        $querylang = clone($query);
        $querylang->where('a.language = '.$this->db->q($language->lang_code));

        $this->db->setQuery($querylang);
        $return += (int) $this->db->loadResult();
      }
    }
    $querylang = clone($query);
    // Get the total number of content items translated languages to index.
    $querylang->where('a.language = '.$this->db->q('*'));
    $this->db->setQuery($querylang);
    $return += ((int) $this->db->loadResult() * count($languages));

    return $return;
  }

[Finarfin]

Almost perfect

Two points


Why the if ($query instanceof JDatabaseQuery) ? I don´t know why, but with that if it calls the parent and so give the bad number. I don´t know why is that the check, but I disable it for testing and worked. Perfectly! :O

Another topic, I guess that this

    $querylang = clone($query);
    // Get the total number of content items translated languages to index.
    $querylang->where('a.language = '.$this->db->q('*'));
    $this->db->setQuery($querylang);
    $return += ((int) $this->db->loadResult() * count($languages));

have to be inside the if (array_key_exists('language', $columns)) {, but is a not really a big deal.




Another thing

Inside the getItems, you have to divide the offset between count(languages). Why? Because you are returning count(language)*number of items, but the table you are visiting is not queried as fast.

Example

Offset = 0
Limit = 60

getItems()
- select from 0 to 60 (for jdiction I will get 180)

Offset = 180
Limit = 60

here, if I do the select from 180 with a 60 limit, I´m leaving behind the entries between 60 and 180 so... I have to divide.

And that´s all for today. I want to clean my code, maybe do some tests more and I´ll be able to try to compilate the patches done so you have them available.

Regards and thanks for your help, that last function, saved me

[Harald Leithner]

You are right this function should work better:

Code: [Select]

protected function getContentCount() {

    $jDiction = jDiction::getInstance();
    $languages = $jDiction->getLanguages(true);

    $return = 0;

    // Get the list query.
    $query = $this->getListQuery();

    // Check if the query is valid.
    if (empty($query)) {
      return $return;
    }

    // we don't support non JDatabaseQueries here atm
    if (!$query instanceof JDatabaseQuery) {
      return parent::getContentCount();
    }

    $columns = $this->db->getTableColumns($this->table);

    if (array_key_exists('language', $columns)) {
      $query = clone($query);
      $query->clear('select')
        ->select('COUNT(*)')
        ->clear('order');

      // Get the total number of content items per language to index.
      foreach($languages as $language) {

        $querylang = clone($query);
        $querylang->where('a.language = '.$this->db->q($language->lang_code));

        $this->db->setQuery($querylang);
        $return += (int) $this->db->loadResult();
      }

      // Get the total number of content items translated languages to index.
      $query->where('a.language = '.$this->db->q('*'));
    }
    $this->db->setQuery($query);
    $return += ((int) $this->db->loadResult() * count($languages));

    return $return;
  }
The Problem with the offset is that onBuildIndex calculates the batchOffset based on the rows it gets and not on the offset it gives me, jDictions returns more rows then expected...

I'm not sure how to solve this. Maybe by changing the onBuildIndex from:

Code: [Select]

// Adjust the offsets.
$offset++;
$iState->batchOffset++;
$iState->totalItems--;

to

Code: [Select]


// after the for loop
$offset += $offset;
$iState->batchOffset += $offset;
$iState->totalItems -= $offset;

but this would fail if we return less $items then offset... maybe changing this to count($items) if $offset is less then count($items).

[Harald Leithner]

Maybe this override could fix the count problem.

Code: [Select]

  public function onBuildIndex()
  {
    // Get the indexer and adapter state.
    $iState = FinderIndexer::getState();
    $aState = $iState->pluginState[$this->context];

    // Check the progress of the indexer and the adapter.
    if ($iState->batchOffset == $iState->batchSize || $aState['offset'] == $aState['total'])
    {
      return true;
    }

    // Get the batch offset and size.
    $offset = (int) $aState['offset'];
    $limit = (int) ($iState->batchSize - $iState->batchOffset);

    // Get the content items to index.
    $items = $this->getItems($offset, $limit);

    // Iterate through the items and index them.
    for ($i = 0, $n = count($items); $i < $n; $i++)
    {
      // Index the item.
      $this->index($items[$i]);

    }

    // Adjust the offsets.
    if ($limit > count($items)) {
      $limit = count($items);
    }
    $offset += $limit;
    $iState->batchOffset += $limit;
    $iState->totalItems -= $limit;

    // Update the indexer state.
    $aState['offset'] = $offset;
    $iState->pluginState[$this->context] = $aState;
    FinderIndexer::setState($iState);

    return true;
  }