RE: Using HiveSQL with Python on Mac

avatar

You are viewing a single comment's thread:

For hive this is the best but unfortunately for second layer tokens I don't think HiveSQL is the solution though since we can't depend on Custom JSON . I used to use HiveSQL , now I am shifitng to hive-engine API.

Posted Using LeoFinance Beta



0
0
0.000
15 comments
avatar

Interesting. I haven’t explored all tables and columns yet. I would think all data should be available there. I will keep that in mind. Thanks.

0
0
0.000
avatar

Well yeah the last table is TxCustoms table where you can see the 2nd layer custom json data .

There are a lot of duplicates over there ,
For ex :

I tried to retrieve total LEO staked data - Actual stake was around 7k ( for around 10 days ) but I got 8.3k . When I check manually there were lot of stake json's appearing twice .

All the hive related data are accurate though . Only the Custom JSON part has irregularities.

Posted Using LeoFinance Beta

0
0
0.000
avatar

Thank you for this information. I will keep that in mind when I explore the data in that table. I think those issues can be resolved though.

  1. If there is error when the custom_json data is stored on HiveSQL, we should probably let @arcange know.
  2. If the problem is due to how layer 2 data is stored on blockchain, I would assume it could be sorted out with python.

It is interesting. I will definitely keep that in mind when I get to learning more about custom_json.

0
0
0.000
avatar

Yeah 2nd point is actually what I used for a while but when the tx's are huge it became increasingly difficult for me to sort out the irregularities .

The problem isn't with HiveSQL . check this out -

image.png

I went to that particular user history on hiveblocks and checked manually and I saw that there was actually two custom_json for the stake . One was rejected and one was accepted but that won't be shown in HiveSQL table , HiveSQL TxCustoms table shows all custom_json tx's irrespective of whether it is valid or not. It is shown in 'logs' in hive explorer that one is accepted and other is rejected.

Posted Using LeoFinance Beta

0
0
0.000
avatar

How are ‘rejected’ and ‘accepted’ stored on the blockchain? separate custom_json?

0
0
0.000
avatar
(Edited)

That's what troubled me too, I asked the same on discord couple of weeks ago and got this answer

There is a bug in Keychain where some transactions are broadcasted twice. I guess this is something related to that. Even though two custom jsons were processed at the same time, Hive Engine would have accepted only one.

Then I headed down to Hive Engine explorer and saw this

image.png
First image ( accepted stake )

image.png
Second image ( rejected stake )

image.png

So the payload for both the custom_json was the same but the 'logs' was different . So if you want reliable 2nd layer token JSON data , according to me , hiveengine API is where to look at.

HiveSQL stores 'payload' in its table but what we need here is actually 'logs' . So right now I am extracting all the data from December 1st and storing it in JSON format ( although I prefer CSV , it will become difficult for me to clean it later) .

Did any of the above make sense?

!WINE

Posted Using LeoFinance Beta

0
0
0.000
avatar

Yes, everything makes sense. I looked at some examples. You are right, there is no way to see which one is rejected, or accepted in HiveSQL.

The question is if 'logs' is actually stored in the blockchain or is it probably stored in a separate hive-engine database?

0
0
0.000
avatar

Hm that's the question I am trying to find answer too. I would love to work with HiveSQL itself because via API it takes a lot of time.

Posted Using LeoFinance Beta

0
0
0.000
avatar

I don’t know if this will help, but one idea is to collect all transaction_ids where logs show ‘error’ from hive-engine api. Then use this list to exclude in HiveSQL query.

0
0
0.000
avatar

For sure that can be done but as far as I have explored till now , you can get the logs data only by going through all the blocks one by one.

Using a for loop , right now it's fetching 1000 blocks in 4 minutes so yeah it takes a long time again.

0
0
0.000
avatar
(Edited)

custom_json are non-consensus operations. This means that the blockchain will never reject any (except if authorities are missing).

Therefore, HiveSQL (like the blockchain) includes all broadcasted custom_json, even if "duplicate" or considered as invalid by a 2nd layer app.

This is why there is no hive-engine "state" table in HiveSQL because the hive-engine code is closed source and HiveSQL doesn't know how to interpret those operations.

0
0
0.000
avatar

Got it :) Thanks for taking your time and clearing it.

0
0
0.000