Modify phash algorithm for short files

[Flashy78]

Resolves #3722

Updates new phash generation to use:

Time	Frames
x > 2.5min	Current 25
2.5min <= x > 1.5m	16
1.5min <= x > 45s	9
x <= 45s	4

Open question on what to do about existing hashes.

1. Add a migration button to generate phashes for all <2.5min videos in a user's collection.
2. Update hash code to allow for multiple phashes, so users can have old and new hashes to tag with. Including modifying tagging code to check all hashes against StashDB.
3. User option to also generate old style hashes.
4. A plugin users can run to upload all newly generated hashes to StashDB for scenes with valid StashIDs. This is probably far too dangerous.
5. Do nothing. Users generating new short hashes will get those and organically populate StashDB with them.

[stg-annon]

I would think that we could keep existing phashes perhaps change the fingerprint type to phash2 then a user may elect to purge existing hashes and regenerate. This way the fingerprint type keeps track of versioning across stash and stash-box

[dingdongsolong4]

imo, I don't think adding an additional phash type or allowing for multiple phashes is a good idea. It's just going to add complexity in many different places.

Phashes aren't really exact, they're only there to aid in matching. Incorrect phashes are an issue in stash-boxes anyway, from users submitting the wrong scene entirely - so I don't think that adding all this additional complexity to try avoid this problem for short files is needed. I think this issue should be dealt with on the stash-box side - "new-style" phashes can overtake the previous ones organically, and if there are issues with specific scenes, the problematic old phashes should be removed and "banned" from stash-box individually.

A migration to regenerate phashes for existing short scenes is a good idea - if it is "advertised" enough then we'd have very few "old-style" phashes submitted once the change makes it to a release. An option to submit the new phashes to stash-boxes would also be useful, but it should very definitely be disabled by default.

A toggle to enable to previous behaviour would be fine, but I don't know why you'd want to actually use it (ie why you'd want the old behaviour).

These are just my thoughts as it stands - I am open to be convinced otherwise.

[WithoutPants]

My proposal is to have a schema migration that removes existing phashes for scenes < 2.5 minutes - with a migration warning indicating that this is what will be done, and that these phashes need to be regenerated. This reduces the amount of people submitting old version phashes to only those running older versions. On the stash-box side, admins could do the same thing (manually via sql) but imo it's not a huge issue.

[Flashy78]

I like getting rid of "old" hashes and just going with the new ones. However there will be a good while where new users who are trying to match content on Stashbox from before the change, will get zero results until someone does a title match and submits the new hash to the existing scene.

Maybe a transition phase?

• Migration warning that <2.5 min videos will need to be rehashed
• Migrate all "old" hashes to a new type phash-old
• During transition phase, generate and save both types of hashes (videos are short so not a big burden)
• In findStashBoxScenesByFingerprints() hardcode phash-old back to phash, so both the new and old are sent to Stashbox as phash, and Stashbox can return matches for either one.
• After a match when the Submit Fingerprints happens, only send the "new" phash
• One or two Stashapp versions later, remove the duplicate phash-old generation and delete them all from the user's db.

Granted it is some extra complexity, but it might allow for Stashboxes to get the new hashes attached to existing scenes quicker than requiring someone to do a title Match to submit them.

[Flashy78]

Implemented my transition phase proposal.

Seems to work well matching on the old hash and submitting the new hash.
Only real downside is generating multiple phashes for short videos, but it's seamless to the user and shouldn't be noticeable for most people.

[Flashy78]

Not sure why the tests fail with the migration, they run fine locally.
Anyone have any ideas?

[dingdongsolong4]

Not sure why the tests fail with the migration, they run fine locally. Anyone have any ideas?

@Flashy78 There are two migration files for schema version 50 - version 50 is already taken by the multiple image URL migration. All you should need to do is change 50 to 51.

[Flashy78]

@Flashy78 There are two migration files for schema version 50 - version 50 is already taken by the multiple image URL migration. All you should need to do is change 50 to 51.

As always, the error message was pretty clear what the issue was. Thanks for helping me realize my fork was a few days old.

[Gykes]

superseded by #7225

This had too many conflicts and old code to be resurrected