The recent switch to Python3 (2ccdf0e396e82dd) introduced a regression
that led to file content no longer being compared:
In compareEntries(), two generators/iterators are created:
sourceInfoList = filter(lambda sourceInfo: …, sourceZip.infolist())
destinationInfoList = filter(lambda destinationInfo: …, destinationZip.infolist())
Few lines later, those are exhausted:
if len(sourceInfoList) != len(destinationInfoList):
Yet another few lines later, the exhausted generator is used again:
for sourceEntryInfo in sourceInfoList:
… # <-- unreachable
This is caused by behavioral differences between Python2 and Python3:
user@z_signal:~$ python2
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = filter(lambda i: i % 2 == 0, [0, 1, 2, 3, 4, 5, 6])
>>> list(f)
[0, 2, 4, 6]
>>> list(f)
[0, 2, 4, 6]
>>>
user@z_signal:~$ python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = filter(lambda i: i % 2 == 0, [0, 1, 2, 3, 4, 5, 6])
>>> list(f)
[0, 2, 4, 6]
>>> list(f)
[]
>>>
- Check for permissions.
- Fix Welsh positional format.
- Remove UIThread restriction.
- Asynchronous method does not need to be restricted to UIThread and there is no StaticFieldLeak to suppress.
- Fix or Ignore New API errors.
- Reduce severity of some errors from L10N.
We observed IOExceptions loading the Contact Discovery IAS KeyStore. We will now throw an AssertionError upon any error
creating the IAS KeyStore, matching the behaviour for creation of the TrustStore used for the main Signal Service. NB: If
this assertion is hit, the user will not even be able to refresh their contacts with the old directory service.
We never make requests to non-whitelisted domains, but there were
situations where some links would redirect to non-whitelisted domains,
which would hit a final failsafe that resulted in a crash.
To prevent this, we detect bad redirects earlier and fail more
gracefully.
Fixes#8796
This will stop instances of the following from occuring in the logs
on SMS migration:
W/SQLiteCompiledSql: Releasing statement in a finalizer. Please ensure
that you explicitly call close() on your cursor: INSERT INTO sms
(address, person, date_sent, date, protocol, read, status, type,
reply_path_present,
net.sqlcipher.database.DatabaseObjectNotClosedException: Application did not close the cursor or database object that was opened here
at net.sqlcipher.database.SQLiteCompiledSql.<init>(SQLiteCompiledSql.java:62)
at net.sqlcipher.database.SQLiteProgram.<init>(SQLiteProgram.java:109)
at net.sqlcipher.database.SQLiteStatement.<init>(SQLiteStatement.java:39)
at net.sqlcipher.database.SQLiteDatabase.compileStatement(SQLiteDatabase.java:1647)
at org.thoughtcrime.securesms.database.SmsDatabase.createInsertStatement(SmsDatabase.java:767)
at org.thoughtcrime.securesms.database.SmsMigrator.migrateConversation(SmsMigrator.java:166)
at org.thoughtcrime.securesms.database.SmsMigrator.migrateDatabase(SmsMigrator.java:210)
at org.thoughtcrime.securesms.service.ApplicationMigrationService$ImportRunnable.run(ApplicationMigrationService.java:159)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
We aren't closing Statement objects before the finalizer on those
objects runs. When the GC runs, we'll get warnings like the above
which alert us to the fact that these objects are being automatically
closed for us in the finalizer, but that this is suboptimal behavior.
If we leave too many Statement (or Cursor) objects to be closed in
their finalizers, when the GC runs, it'll take longer than 10 seconds
to close them all and Android will kill the app. This 10 second limit
is hardcoded and we can only try to avoid it. A crash will look like:
java.util.concurrent.TimeoutException: net.sqlcipher.database.SQLiteCompiledSql.finalize() timed out after 10 seconds
at java.lang.Object.wait(Native Method)
at java.lang.Thread.parkFor$(Thread.java:1220)
at sun.misc.Unsafe.park(Unsafe.java:299)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:810)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:844)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1173)
at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:196)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:257)
at net.sqlcipher.database.SQLiteDatabase.lock(SQLiteDatabase.java:553)
at net.sqlcipher.database.SQLiteCompiledSql.releaseSqlStatement(SQLiteCompiledSql.java:106)
at net.sqlcipher.database.SQLiteCompiledSql.finalize(SQLiteCompiledSql.java:152)
at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:202)
at java.lang.Daemons$FinalizerDaemon.run(Daemons.java:185)
at java.lang.Thread.run(Thread.java:818)
I was able to replicate the above crash consistently on a
Samsung Galaxy S7 edge when importing well over 100k SMS messages.
But as soon as I attached a debugger the crash did not persist. I
assume this is because of some VM-level interactions between the two
and did not investigate further after fixing it.
I do not have access to the stack trace for issue #7953 but this
could potentially resolve it. The crash is identical to that in #7477
but this patch is for SMS migration not restoring from a backup. I
was not able to replicate the crash on restoring a >100k message
backup.
