tools/hiccup/hiccup-server.git
21 months agoUpdate README.md on linters
Mitja Nikolaus [Thu, 29 Nov 2018 13:58:55 +0000 (14:58 +0100)]
Update README.md on linters

Issue: HIC-276
Change-Id: I8ce2b175e54260320c6ca5544145c61a66236bb8

21 months agoMake pylint pre-commit hook strict
Mitja Nikolaus [Thu, 29 Nov 2018 13:50:42 +0000 (14:50 +0100)]
Make pylint pre-commit hook strict

Issue: HIC-275
Change-Id: I2c0b3ae00b850f16faf632ca0105c0d177d257e4

21 months agoLock affected rows when assigning local IDs
Mitja Nikolaus [Tue, 13 Nov 2018 16:12:51 +0000 (17:12 +0100)]
Lock affected rows when assigning local IDs

Lock the targeted device when calculating the next heartbeat and crash
report local IDs. Lock the targeted crash report when calculating the
next log file local ID.

This change prevents race conditions that have occured previously.

Issue: HIC-251
Change-Id: I80a8498849e20769b0ab36afe68e6c20b3108c6d

21 months agoAdd test cases for local ID assignment race conditions
Mitja Nikolaus [Tue, 13 Nov 2018 15:48:20 +0000 (16:48 +0100)]
Add test cases for local ID assignment race conditions

Issue: HIC-251
Change-Id: I9616b786a44bba2621a8a800035ff1189d3d9b19

21 months agoEnable and fix skipped test cases
Mitja Nikolaus [Wed, 7 Nov 2018 10:57:02 +0000 (11:57 +0100)]
Enable and fix skipped test cases

Add manual calls to flush the database inside test functions to prevent
creation of duplicate entries.

Issue: HIC-180
Change-Id: I3902fb5289becdaf8baae089801d3afbbe5aa52e

21 months agoDisallow duplicate heartbeats and crashreports
Mitja Nikolaus [Wed, 7 Nov 2018 10:53:59 +0000 (11:53 +0100)]
Disallow duplicate heartbeats and crashreports

Add unique constraints and corresponding schema and data migration.
Adapt all test cases so that only unique heartbeats and crashreports
are sent. Delete test cases that are inappropriate as no duplicate
entries can exist in the database anymore.

Issue: HIC-180
Change-Id: I768d1610d4482c9d61b76cdbc588334198bfe415

21 months agoAdd logging configuration for all apps
Mitja Nikolaus [Fri, 23 Nov 2018 12:47:22 +0000 (13:47 +0100)]
Add logging configuration for all apps

Issue: HIC-272
Change-Id: Iab24866507914b76a611537b166cfb050ce472d4

21 months agoDelete log files on instance deletion
Mitja Nikolaus [Thu, 22 Nov 2018 15:40:15 +0000 (16:40 +0100)]
Delete log files on instance deletion

Delete log files from the file system when the corresponding model
instance is deleted. Add a corresponding test case.

Issue: HIC-271
Change-Id: I44709c78839091272ccc9b931e7cbe7b3947dff9

21 months agoMerge common code of dummy testing classes
Mitja Nikolaus [Tue, 27 Nov 2018 13:57:39 +0000 (14:57 +0100)]
Merge common code of dummy testing classes

Move all shared functionality to crashreports/tests/utils.py and make
the dummy in crashreport_stats/tests/test_rest_endpoints.py inherit from
the crashreports dummy class to prevent code duplication.

Issue: HIC-269
Change-Id: I69ce569c13169ded5857638c466bea8a650f38fd

21 months agoDisallow API access for non-Fairphone-staff users
Mitja Nikolaus [Mon, 5 Nov 2018 09:22:22 +0000 (10:22 +0100)]
Disallow API access for non-Fairphone-staff users

Do not give permissions for users that are not in the Fairphone
staff group.

Issue: HIC-250
Change-Id: I400eade07396d24e6168ae12561a643a403b8491

21 months agoReturn permission denied instead of redirecting in stats views
Mitja Nikolaus [Wed, 21 Nov 2018 11:14:45 +0000 (12:14 +0100)]
Return permission denied instead of redirecting in stats views

Return a 403 permission denied error if a user is not part of the
Fairphone staff group. This breaks the infinite redirect loop.

Issue: HIC-260
Change-Id: I682127c2165f826257a84b0bf246fc9fb86813ea

22 months agoUse Fairphone staff client instead of admin user for tests
Mitja Nikolaus [Mon, 5 Nov 2018 09:00:53 +0000 (10:00 +0100)]
Use Fairphone staff client instead of admin user for tests

Run tests against the REST API using a client that is related to a
Fairphone staff user instead of an admin user.

Issue: HIC-249
Change-Id: I33666235ff186fef566bdb0aa6e7a9c768856949

22 months agoDisable stats access for device owners
Mitja Nikolaus [Mon, 12 Nov 2018 18:07:30 +0000 (19:07 +0100)]
Disable stats access for device owners

Allow only Fairphone staff group members to access stats functions.

Issue: HIC-203
Change-Id: Ic989ae2d0d7a4b68cd57b114e91e2b14c52227fc

22 months agoReplace raw SQL queries with native Django code
Mitja Nikolaus [Fri, 2 Nov 2018 16:10:02 +0000 (17:10 +0100)]
Replace raw SQL queries with native Django code

Issue: HIC-248
Change-Id: I46542f2b6558f6ad3e372cb2900fbcf8614a3144

22 months agoAdd fallback for the last_active date for devices without heartbeats
Mitja Nikolaus [Fri, 2 Nov 2018 14:30:45 +0000 (15:30 +0100)]
Add fallback for the last_active date for devices without heartbeats

Return the board date of the device as fallback.

Issue: HIC-206
Change-Id: I58090e9448aedf1f0dfac4ea0b39f11b941ed109

22 months agoExtract get_object_by_lookup_fields method
Mitja Nikolaus [Fri, 2 Nov 2018 14:07:20 +0000 (15:07 +0100)]
Extract get_object_by_lookup_fields method

Issue: HIC-178
Change-Id: I406ce8b8ca0b5dbec587a6786f820746240bc835

22 months agoAdd manage.py to the package
Mitja Nikolaus [Thu, 15 Nov 2018 08:59:32 +0000 (09:59 +0100)]
Add manage.py to the package

Add the Django manage.py script to the package so that it can be used
when installing the server without checking out the code.

Issue: HIC-264
Change-Id: I8f1b70fd1238f909988d7cfcde4a9ef0b3f81ca4

22 months agoInclude HTML, CSS and JS files in package
Mitja Nikolaus [Fri, 9 Nov 2018 16:11:05 +0000 (17:11 +0100)]
Include HTML, CSS and JS files in package

Issue: HIC-262
Change-Id: I274d3f7d9277356c436d8c09c275a6187037d726

22 months agoAdd missing __init__.py files
Mitja Nikolaus [Fri, 9 Nov 2018 16:07:06 +0000 (17:07 +0100)]
Add missing __init__.py files

Issue: HIC-261
Change-Id: Ie4fa2d3f66d75dcda0c4afdaaafcc38a2b8411c4

22 months agoAdd Apache 2.0 license
Mitja Nikolaus [Wed, 14 Nov 2018 09:55:23 +0000 (10:55 +0100)]
Add Apache 2.0 license

Issue: HIC-263
Change-Id: I745ede74b85d8c60566bab4ad90a69f134b3f456

22 months agoAdd migration for creation of Fairphone staff group
Mitja Nikolaus [Wed, 17 Oct 2018 12:13:06 +0000 (14:13 +0200)]
Add migration for creation of Fairphone staff group

Issue: HIC-247
Change-Id: I44b3db32e3646fae8d7d9ff37dc203c6c2c65756

22 months agoMove uploads directory configuration to storage API
Franz-Xaver Geiger [Tue, 9 Oct 2018 12:52:26 +0000 (14:52 +0200)]
Move uploads directory configuration to storage API

Make use of the MEDIA_ROOT setting to store logfiles for tests in
dedicated test directories. Further adapt the logfile path migration to
the use of the storage API.

Issue: HIC-240
Change-Id: I6dd3048f9987f1185fe124f673a0b0ac7ac101e1

23 months agoAdd author and update version in setup.py
Mitja Nikolaus [Mon, 15 Oct 2018 11:37:46 +0000 (13:37 +0200)]
Add author and update version in setup.py

Issue: HIC-230
Change-Id: Ic85f9d2b5fa619645b5e0a00a2dc6032287cdcd9

23 months agoAdd setup.py
Mitja Nikolaus [Fri, 12 Oct 2018 12:18:06 +0000 (14:18 +0200)]
Add setup.py

Add setup.py file for proper installation. Move dependency declarations
from the requirements.txt file into the setup.py to avoid duplicate
information. Make tox use the setup.py for creating test environments.

Issue: HIC-230
Change-Id: I260f0ee841f1bb00714fc20dcf352fbc4e80e25a

23 months agoAdd migration for new logfile paths
Mitja Nikolaus [Fri, 5 Oct 2018 07:58:18 +0000 (09:58 +0200)]
Add migration for new logfile paths

Update all logfile paths in the database and migrate the referenced log
files to the new directory structure.

Issue: HIC-238
Change-Id: I69b53993e77079258e12ec995cbf85d3eb0a780b

23 months agoAdd a teardown method to LogfileUploadTest
Mitja Nikolaus [Thu, 4 Oct 2018 13:17:48 +0000 (15:17 +0200)]
Add a teardown method to LogfileUploadTest

Add method to delete the created file for the log file upload test.

Issue: HIC-235
Change-Id: Id7db99cebad38d50af0082809bedd07168c38c6e

23 months agoImprove test for log file upload
Mitja Nikolaus [Thu, 4 Oct 2018 13:09:11 +0000 (15:09 +0200)]
Improve test for log file upload

Validate that the log file has actually been created and that its
contents are correct.

Issue: HIC-236
Change-Id: I7586ab5bba2a7baa5a41085c4aab64f4f1df1314

23 months agoAdapt path where log files are stored
Mitja Nikolaus [Thu, 4 Oct 2018 13:03:31 +0000 (15:03 +0200)]
Adapt path where log files are stored

Split up the directory named after the device UUID into subdirectories
to decrease the number of directories in the first level.

Issue: HIC-237
Change-Id: Ifc33728553e515f6efeb2e12ad11c7e2d6eaf181

23 months agoUse compressed log file for upload test
Mitja Nikolaus [Thu, 4 Oct 2018 09:15:29 +0000 (11:15 +0200)]
Use compressed log file for upload test

The actual client sends zipped log files to the server, thus the test
should be aligned to do it the same way.

Issue: HIC-234
Change-Id: I76831113e3f98a3676784f59d4ccd55649049451

23 months agoAdd tests for updating the stats version counters twice
Mitja Nikolaus [Mon, 24 Sep 2018 13:00:04 +0000 (15:00 +0200)]
Add tests for updating the stats version counters twice

The tests ensure that information from old and new report counters is
correctly integrated when running the stats update command twice.

Issue: HIC-150
Change-Id: I8db55fa3d30d1a15446184be9f71efd1e805217d

23 months agoAdd tests for the stats reset command
Mitja Nikolaus [Mon, 1 Oct 2018 13:15:04 +0000 (15:15 +0200)]
Add tests for the stats reset command

Add test cases to check whether unrelated versions are deleted and
counters are appropriately updated.

Issue: HIC-152
Change-Id: I4dc151f8990b0a5e09f0b9b4337580d07f54332a

23 months agoRefactor dummy version creation method
Mitja Nikolaus [Mon, 1 Oct 2018 13:10:18 +0000 (15:10 +0200)]
Refactor dummy version creation method

Add argument to be able to create both Version and RadioVersion
instances with the same method.

Issue: HIC-221
Change-Id: I0cc4d5fe658c4f15a650281613ba8f02718eafbd

23 months agoAdd tests for creating stats from reports of different devices
Mitja Nikolaus [Fri, 21 Sep 2018 13:59:54 +0000 (15:59 +0200)]
Add tests for creating stats from reports of different devices

These test cases are more realistic as the counters have to be
calculated by accumulating the reports from different devices.

Issue: HIC-184
Change-Id: I9e18d58f66c4eb8d6dc2fb44be908ef1feff382b

23 months agoRefactor crashreport stats view permissions
Mitja Nikolaus [Fri, 14 Sep 2018 13:33:29 +0000 (15:33 +0200)]
Refactor crashreport stats view permissions

Align authorization permissions with the rest of the stats API:
Not only users of the correct group are authorized but also admin
users.

This change fixes some tests that now do not need to be skipped
anymore.

Issue: HIC-204
Change-Id: If82b9156e84f78f8dbe7c1a930c7bf8c3941e7e2

23 months agoAdd tests for the crashreport stats template tags
Mitja Nikolaus [Fri, 21 Sep 2018 12:54:26 +0000 (14:54 +0200)]
Add tests for the crashreport stats template tags

Test rendering of the custom template tags.

Issue: HIC-207
Change-Id: I746637e165f04129b33bcbd43085607dba19a55a

23 months agoAdd tests for the allauth social account adapters
Mitja Nikolaus [Fri, 21 Sep 2018 08:55:40 +0000 (10:55 +0200)]
Add tests for the allauth social account adapters

Issue: HIC-196
Change-Id: Iec0a5f732c2c03339f389ce3bf2975cd58d148a4

23 months agoRestrict coverage reports to local python files
Franz-Xaver Geiger [Thu, 20 Sep 2018 15:41:27 +0000 (17:41 +0200)]
Restrict coverage reports to local python files

Coverage reports include information on general python source (e.g. when
run in a pyenv environment). This change limits coverage recordings to
the local path.

Issue: HIC-239
Change-Id: I02e97d2de54d13bf20b33f59d1377fe79a67677c

2 years agoAllow tox to pass through arguments
Franz-Xaver Geiger [Wed, 19 Sep 2018 08:27:04 +0000 (10:27 +0200)]
Allow tox to pass through arguments

Passing through commandline arguments to commands allows to easily
configure what is run in environments. For instance, this allows to
specify what particular files to run linters against or what tests to
execute.

This change additionally moves arguments to linter commands into a
configuration constant.

Issue: HIC-213
Change-Id: Id6c38cd8376ef505dd372da1ee13edb289a10974

2 years agoRestructure tests into separate modules
Mitja Nikolaus [Tue, 18 Sep 2018 15:50:15 +0000 (17:50 +0200)]
Restructure tests into separate modules

Move all tests to a designated subdirectory and split the modules up
so that each test module covers one module.

Further rename some misnamed Test classes and adapt the test coverage
configuration so that the new test modules are omitted in the reports.

Issue: HIC-212
Change-Id: Idfa309aad34b8adae181cc76e95b67a86231bc69

2 years agoAdd tests for stats endpoints authorization
Mitja Nikolaus [Tue, 18 Sep 2018 14:39:46 +0000 (16:39 +0200)]
Add tests for stats endpoints authorization

Add tests that ensure all stats endpoints URLs are only accessible
for admin users and Fairphone staff.

Further move attributes from _VersionTestCase to a level where they are
actually used, i.e. VersionTestCase and VersionDailyTestCase.

Issue: HIC-205
Change-Id: Iad0dc2acd7d8eb5d33c6681e07b6bbbbdcb28d94

2 years agoAdd tests for statistics views
Mitja Nikolaus [Fri, 14 Sep 2018 13:40:08 +0000 (15:40 +0200)]
Add tests for statistics views

Issue: HIC-189
Change-Id: I1eb2aec3d549c88d83c1f7365ebb01aab939d822

2 years agoAdd tests for the status endpoint
Mitja Nikolaus [Wed, 5 Sep 2018 11:45:36 +0000 (13:45 +0200)]
Add tests for the status endpoint

Issue: HIC-188
Change-Id: Ia0d0127ea706154c6bb1d2cd8f199a655ffc4a5d

2 years agoFix parameter name in urls.py
Mitja Nikolaus [Wed, 5 Sep 2018 10:33:14 +0000 (12:33 +0200)]
Fix parameter name in urls.py

Requesting logfiles did not work because of a misalignment of the
parameter names in urls.py and rest_endpoint.py.

Issue: HIC-191
Change-Id: I6e2806357069009802eab13a0eece94d12f05b97

2 years agoAdd tests for device statistics
Mitja Nikolaus [Wed, 5 Sep 2018 10:18:35 +0000 (12:18 +0200)]
Add tests for device statistics

Add test cases for device stats, device report history, device update
history and downloading log files.

Issue: HIC-188
Change-Id: I1c9777fadf12c210d1e564c9ff1718afb9ea7f23

2 years agoDrop support for SQLite databases
Mitja Nikolaus [Wed, 5 Sep 2018 14:05:39 +0000 (16:05 +0200)]
Drop support for SQLite databases

The production server uses PostgreSQL. No other databases are supported
and thus the support for SQLite is dropped. Consequently, the
migrate_sqlite2psql tool is also not needed anymore and is deleted.

Issue: HIC-192
Change-Id: I47f811c1d5ce81a8856b36ce9f6cf7b2be5da0f4

2 years agoAdd tests for updating the stats released_on field
Mitja Nikolaus [Tue, 4 Sep 2018 12:23:19 +0000 (14:23 +0200)]
Add tests for updating the stats released_on field

Test that the 'released_on' field changes or not once an older report
has been sent depending on whether the field has been manually changed.

Issue: HIC-182
Change-Id: I443fac6dea453dcde96f1c0ead4d25153a729198

2 years agoAdd tests for false duplicates
Mitja Nikolaus [Tue, 4 Sep 2018 10:17:45 +0000 (12:17 +0200)]
Add tests for false duplicates

Add test cases for reports that have the same timestamp but come from
different devices. These reports are not counted at the moment due to
a bug in the deduplication procedure. Thus, the tests are skipped for
now.

Issue: HIC-181
Change-Id: I1bf107fdeed2287d1c57a999e6de7d316d232074

2 years agoUpdate requirements.txt
Mitja Nikolaus [Tue, 4 Sep 2018 08:30:03 +0000 (10:30 +0200)]
Update requirements.txt

Delete unused requirements and add missing requirements.

Issue: HIC-186
Change-Id: I30258831a09347973d2456c1cbe2b223e4243c0e

2 years agoRename misnamed class
Mitja Nikolaus [Mon, 3 Sep 2018 09:56:31 +0000 (11:56 +0200)]
Rename misnamed class

Rename ListCreateView to ListView in logfiles REST API as it is not
capable of anything other then listing since it inherits from
ListAPIView.

Issue: HIC-185
Change-Id: I3604fb96d7f4a1889df6dc6dbfd4b69fbd147746

2 years agoUpdate .gitignore
Mitja Nikolaus [Mon, 3 Sep 2018 09:40:33 +0000 (11:40 +0200)]
Update .gitignore

Update .gitignore with latest version from
https://github.com/github/gitignore/blob/master/Python.gitignore

Further add Pycharm project settings folder and delete duplicate lines.

Issue: HIC-53
Change-Id: Id2b2fb25f256eed9cf3ace7d3969f5f123297a87

2 years agoDelete obsolete documentation file
Mitja Nikolaus [Thu, 30 Aug 2018 08:31:46 +0000 (10:31 +0200)]
Delete obsolete documentation file

The file was incomplete and its information is replaced by the
automated documentation in api-endpoints.md and the wiki.

Issue: HIC-171
Change-Id: I54ecec5d65537b634de15cbc01c383e2cb7f4cbe

2 years agoAdd documentation for authentication methods
Mitja Nikolaus [Tue, 28 Aug 2018 13:31:29 +0000 (15:31 +0200)]
Add documentation for authentication methods

Add swagger security definitions to settings and security parameters
to method annotations. Update the automated documentation file.

Issue: HIC-171
Change-Id: I34610f98186fb04664855ecbcfa3736b9c59c6c0

2 years agoLog only exceptions that are unexpected
Mitja Nikolaus [Thu, 13 Sep 2018 09:15:18 +0000 (11:15 +0200)]
Log only exceptions that are unexpected

ObjectDoesNotExist and TypeError exceptions are expected to be raised
and thus does not need to be logged.

Issue: HIC-199
Change-Id: Ic4d305f72d69dfb6929047a6c7be7ff7f1e54a92

2 years agoSet up test coverage tool
Mitja Nikolaus [Wed, 29 Aug 2018 10:31:57 +0000 (12:31 +0200)]
Set up test coverage tool

Introduce two new commands for tox to measure test coverage and
generate reports using coverage.py

Issue: HIC-162
Change-Id: Iaaf77d981a335e1befcfb59e2294250ed22e0e2b

2 years agoUpdate method signatures to match signatures of superclasses
Mitja Nikolaus [Fri, 31 Aug 2018 08:55:43 +0000 (10:55 +0200)]
Update method signatures to match signatures of superclasses

Issue: HIC-170
Change-Id: Iaad395237615dde9fb76e78ea619e89d40a8102e

2 years agoImplement abstract method of superclass
Mitja Nikolaus [Fri, 31 Aug 2018 08:50:55 +0000 (10:50 +0200)]
Implement abstract method of superclass

Since PrivateField is read only, there will never be any incoming data,
thus we raise NotImplementedError in the transformation method.

Issue: HIC-170
Change-Id: Iebc423934600c6c307c53a1fc37d103a2023743f

2 years agoRename misnamed variables
Mitja Nikolaus [Thu, 30 Aug 2018 15:29:27 +0000 (17:29 +0200)]
Rename misnamed variables

Issue: HIC-170
Change-Id: I610eb79a47135698ab49c1fc2c0c6963ff1fe302

2 years agoRemove FIXME and TODO comments
Mitja Nikolaus [Fri, 31 Aug 2018 09:17:43 +0000 (11:17 +0200)]
Remove FIXME and TODO comments

Jira issues have been created for each FIXME and TODO comment:

crashreport_stats/management/commands/stats.py
260: Agressively drop duplicates (HIC-180)

crashreport_stats/tests.py
561: Test for false duplicates: same timestamps but different UUIDs (HIC-181)
562:
    Test that the 'released_on' field changes or not once an older
    report has been sent depending on whether the field has been manually
    changed (HIC-182)
565: Test that tests the daily version stats (HIC-183)
566: Test creating stats from reports of different devices/users (HIC-184)

crashreports/models.py
137: Remove logfile_type or make it meaningful (HIC-179)

crashreports/rest_api_heartbeats.py
15:
    There is quite some code duplciation between here and the corresponding
    crashreport code. We should revisit this later. (HIC-178)

Issue: HIC-170
Change-Id: I9c81e34df22ff3d794f209cadbc4be81bdfd6a7d

2 years agoRemove unused format parameters
Mitja Nikolaus [Fri, 31 Aug 2018 08:53:28 +0000 (10:53 +0200)]
Remove unused format parameters

Issue: HIC-170
Change-Id: Id4fb91607abfb38f058e4efa94e830a3e31d6ac7

2 years agoUse more pythonic way of checking whether list is empty
Mitja Nikolaus [Thu, 30 Aug 2018 15:38:51 +0000 (17:38 +0200)]
Use more pythonic way of checking whether list is empty

Issue: HIC-170
Change-Id: I7dbcd429aeacb194e5420ed71b6eb371f67d99d6

2 years agoRemove unneccesary else statements after return
Mitja Nikolaus [Thu, 30 Aug 2018 15:16:21 +0000 (17:16 +0200)]
Remove unneccesary else statements after return

Issue: HIC-170
Change-Id: I88365341e93114e0421922292198fce91c8d48aa

2 years agoReplace bare except statement
Mitja Nikolaus [Thu, 30 Aug 2018 15:09:04 +0000 (17:09 +0200)]
Replace bare except statement

Replace bare except with Exception and add a statement to log the
exception.

Issue: HIC-170
Change-Id: Ieaaff28b824d29e6ef14ebca5e4b97a97e3139be

2 years agoFix too long lines
Mitja Nikolaus [Thu, 30 Aug 2018 14:53:21 +0000 (16:53 +0200)]
Fix too long lines

Adapt line length where possible and add exception for flake8 for long
SQL query lines to keep them readable.

Issue: HIC-170
Change-Id: I9321b514c92814ca9c403e7654cb5ef48dcd8ea4

2 years agoOrganize imports
Mitja Nikolaus [Thu, 30 Aug 2018 14:40:38 +0000 (16:40 +0200)]
Organize imports

Remove unused imports and organize misordered imports.

Issue: HIC-170
Change-Id: I0cb2875a6e4c0a1c4a60740c4161aa2445f4317b

2 years agoReplace crashreport admin management classes with register calls
Mitja Nikolaus [Thu, 30 Aug 2018 14:14:14 +0000 (16:14 +0200)]
Replace crashreport admin management classes with register calls

ModelAdmin classes are not needed if no custom values are defined:
https://docs.djangoproject.com/en/1.11/ref/contrib/admin/#modeladmin-objects

Issue: HIC-195
Change-Id: I9838c01f3d7f9deeff82e63545783a5c87eab164

2 years agoAdd missing docstrings
Mitja Nikolaus [Thu, 30 Aug 2018 12:35:29 +0000 (14:35 +0200)]
Add missing docstrings

Add missing docstrings to modules, classes and functions. Refactor
existing docstrings that are not correctly formatted.

Issue: HIC-170
Change-Id: I766830259b897f8c2d931f3b53ce935e82b161fd

2 years agoRemove unnecessary TEMPLATE_LOADERS setting
Mitja Nikolaus [Thu, 30 Aug 2018 13:18:34 +0000 (15:18 +0200)]
Remove unnecessary TEMPLATE_LOADERS setting

The TEMPLATE_LOADERS are set to the default value and thus can safely
be removed. This clears a warning that was raised on startup.
https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/#upgrading-templates-to-django-1-8

Issue: HIC-165
Change-Id: I40d940669dfffbe619e15aad43170ef03c55a973

2 years agoDelete unused code and empty file
Mitja Nikolaus [Thu, 30 Aug 2018 13:06:27 +0000 (15:06 +0200)]
Delete unused code and empty file

The util.py functions have been replaced by the stats management
command. The 'fill_in_build_fingerprints' function of raw_queries.py
is never called.

Issue: HIC-170
Change-Id: Ifc46582fb08fd6462f2cafbca0a47c82692f2e15

2 years agoDelete all apps.py files
Mitja Nikolaus [Thu, 30 Aug 2018 13:01:37 +0000 (15:01 +0200)]
Delete all apps.py files

Project specific configuration is not being used. The files are
superfluous.

Issue: HIC-170
Change-Id: Ic783722ae9945e51984ff4c95385f13aef808cb5

2 years agoAdd documentation for crashreport_stats endpoints
Mitja Nikolaus [Fri, 24 Aug 2018 14:32:58 +0000 (16:32 +0200)]
Add documentation for crashreport_stats endpoints

Add method annotations and update the automated documentation
file.

Issue: HIC-169
Change-Id: I2823725fe3fbd486bb6472033209aaba4accc9ac

2 years agoAdd documentation for crashreports, heartbeats and logfiles endpoints
Mitja Nikolaus [Tue, 21 Aug 2018 14:20:07 +0000 (16:20 +0200)]
Add documentation for crashreports, heartbeats and logfiles endpoints

Add docstrings and method annotations as well as perform some cleanup
to match the expectations of the linters. Further update the automated
documentation file.

Issue: HIC-168
Change-Id: Iee65c44e93a056f3a6ef00ffa98d945b3401a85f

2 years agoUse type hints instead of type declarations in docstrings
Mitja Nikolaus [Wed, 22 Aug 2018 09:17:50 +0000 (11:17 +0200)]
Use type hints instead of type declarations in docstrings

Some type declarations in docstrings were outdated. These are fixed
and further all declarations are replaced by type hints.

Google style guide promotes the use of type annotated code:
https://github.com/google/styleguide/blob/gh-pages/pyguide.md#221-type-annotated-code

Upcoming commits should include type hints to be able to make use of
static type checkers as for example mypy.

Issue: HIC-174
Change-Id: I00462d4c6d32ba39e0c7ae9a17aaea98dac9b973

2 years agoDisable warning in pylint that causes false positives
Mitja Nikolaus [Fri, 24 Aug 2018 08:39:41 +0000 (10:39 +0200)]
Disable warning in pylint that causes false positives

Disable C0330 as there are currently some false positives:
https://github.com/PyCQA/pylint/issues/289

Issue: HIC-175
Change-Id: I0f6d9d446dffc72c7ea3989fe59b1adfdcc8acbf

2 years agoEnforce pylint using a pre-commit hook
Mitja Nikolaus [Tue, 21 Aug 2018 12:12:01 +0000 (14:12 +0200)]
Enforce pylint using a pre-commit hook

Issue: HIC-149
Change-Id: I543c015bbbd7f36b4188d7bd4ef7dbe9bd8dc1d2

2 years agoRefactor long strings to improve formatting
Mitja Nikolaus [Thu, 23 Aug 2018 16:15:01 +0000 (18:15 +0200)]
Refactor long strings to improve formatting

Merge string that fits on one line and split up another string to
increase consistency.

Issue: HIC-161
Change-Id: Icc713247f3bacb61d6229dc497bba16b9a72ef7b

2 years agoFormat all python files using the formatter
Mitja Nikolaus [Fri, 24 Aug 2018 11:54:48 +0000 (13:54 +0200)]
Format all python files using the formatter

Run `git ls-files '*.py' | xargs black`

Issue: HIC-161
Change-Id: I1619e6296bc4036504c5bb73128f769a1b7b688d

2 years agoSet up black formatter
Mitja Nikolaus [Tue, 21 Aug 2018 08:42:19 +0000 (10:42 +0200)]
Set up black formatter

Add black formatter and a corresponding pre-commit hook.

Issue: HIC-161
Change-Id: I8aed563114a6350cc5803f0c5232d3a20d6d6eb6

2 years agoSet max line length to 80
Mitja Nikolaus [Tue, 21 Aug 2018 15:51:09 +0000 (17:51 +0200)]
Set max line length to 80

Configure the linters to accept a line length of 80 characters,
in line with the Google coding style.

Issue: HIC-172
Change-Id: Ie273abab3326c59b4c88739e6b6d7760bc656759

2 years agoAdd reference to Google coding style in README
Mitja Nikolaus [Mon, 20 Aug 2018 12:26:02 +0000 (14:26 +0200)]
Add reference to Google coding style in README

Issue: HIC-166
Change-Id: I478b686ddc4fdd3866e70f81009d18bcbd85175e

2 years agoUpdate generated API documentation file
Mitja Nikolaus [Mon, 30 Jul 2018 12:23:16 +0000 (14:23 +0200)]
Update generated API documentation file

Update the documentation using the command described in README.md

Issue: HIC-101
Change-Id: I19445c4b171e270433bc2d30074a871d04121996

2 years agoAdd documentation for devices REST API
Mitja Nikolaus [Mon, 30 Jul 2018 12:22:09 +0000 (14:22 +0200)]
Add documentation for devices REST API

Add docstrings and method annotations as well as perform some
minor cleanup.

Issue: HIC-168
Change-Id: I737cda9a935b77f415d191e61563199e49e80dd0

2 years agoAdd drf_yasg app for creating automated documentation
Mitja Nikolaus [Mon, 30 Jul 2018 12:10:27 +0000 (14:10 +0200)]
Add drf_yasg app for creating automated documentation

Further cleanup code to match the expectations of the linters and
add tox commands for building the documentation.

Issue: HIC-167
Change-Id: I816a578bc7995f4560c19003d0cc907a22e740b6

2 years agoUpdate django to 1.11.14
Mitja Nikolaus [Mon, 30 Jul 2018 11:36:14 +0000 (13:36 +0200)]
Update django to 1.11.14

Update django to 1.11.14 LTS and related packages to latest
versions.

For the new version, the following changes had to be done:

  - timezone.UTC() was replaced by timezone.utc

  - rest_framework.filters.FilterSet was replaced by
  django_filters.rest_framework.FilterSet
  (http://www.django-rest-framework.org/topics/3.6-announcement/#djangofilterbackend)

  - rest_framework.filters.DjangoFilterBackend was replaced by
  django_filters.rest_framework.DjangoFilterBackend
  (http://www.django-rest-framework.org/topics/3.6-announcement/#djangofilterbackend)

  - django_filters.rest_framework.DateFilter which is now used
  has a 'field_name' parameter instead of a 'name' parameter
  which was used in django_filters.DateFilter

  - ModelSerializers need an explicit call for fields = '__all__'
  (http://www.django-rest-framework.org/api-guide/serializers/#specifying-which-fields-to-include)

Further, the affected code has been cleaned up and documented to
match the expectations of the linters.

Issue: HIC-164
Change-Id: Icf10d211860f7f055a63b9b8e680832753a0e149

2 years agoDescribe command to add users to group
Franz-Xaver Geiger [Wed, 25 Jul 2018 16:26:54 +0000 (18:26 +0200)]
Describe command to add users to group

Change-Id: I27b7bda66b80da80afc39991226cddd10868a100

2 years agoFix: 'stats reset' command fails with AttributeError
Franz-Xaver Geiger [Tue, 7 Aug 2018 09:51:51 +0000 (11:51 +0200)]
Fix: 'stats reset' command fails with AttributeError

delete_stats() returns count_per_model from
self.stats_model.objects.all().delete(). Keys of that dict are set to
model._meta.label in Collector.delete() (called by QuerySet.delete()).

Further, add a matching test case.

Issue: HIC-151
Change-Id: I96838820cb7f8fd7f32825acc1cb52845484bb6a

2 years agoAdd tests for the stats commands debug output
Borjan Tchakaloff [Thu, 22 Mar 2018 14:24:56 +0000 (18:24 +0400)]
Add tests for the stats commands debug output

Issue: HIC-112
Change-Id: I73d5fa62fcf9e4f5bc9c86ee1306c13cf2cbad71

2 years agoFix migrations for statsmetadata model
Mitja Nikolaus [Thu, 16 Aug 2018 07:54:11 +0000 (09:54 +0200)]
Fix migrations for statsmetadata model

Remove manually added 'default=None' that is not needed.

Change-Id: I31ed3e23bb634a7d5c2f2e69c533498815e3491d

2 years agoDelete obsolete views.py
Mitja Nikolaus [Wed, 8 Aug 2018 07:39:31 +0000 (09:39 +0200)]
Delete obsolete views.py

All views of the crashreports application that are in use are
defined in the respective rest_api_*.py files. Classes of the
views.py file are not used anywhere.

Change-Id: I56907c0a2accde76c5ffc61e923c7a8ca6f21038

2 years agoAdd tests to validate ignoring duplicate timestamps
Borjan Tchakaloff [Tue, 20 Mar 2018 13:32:37 +0000 (17:32 +0400)]
Add tests to validate ignoring duplicate timestamps

Issue: HIC-112
Change-Id: I43c9d7d92ca5a73044be7d92710699af4c07f621

2 years agoAdd tests for statistical database computation
Borjan Tchakaloff [Mon, 19 Mar 2018 13:23:58 +0000 (17:23 +0400)]
Add tests for statistical database computation

Issue: HIC-112
Change-Id: I05cac4639f542c22e91dc603035c784b56f57932

2 years agoAdd tests for statistical database REST API
Mitja Nikolaus [Thu, 9 Aug 2018 09:11:28 +0000 (11:11 +0200)]
Add tests for statistical database REST API

Issue: HIC-112
Change-Id: I5d6da6f28f1be7e74fd39af08f79d5cb92d0e6c5

2 years agoRefactor device registration in test cases
Mitja Nikolaus [Fri, 3 Aug 2018 11:55:30 +0000 (13:55 +0200)]
Refactor device registration in test cases

Device registration and admin user creation should only happen
where it is needed. Further update the coding style and add
docstrings to match the expectations of the linters.

Issue: HIC-153
Change-Id: Ic4c6d11f7275411ec1bc0164d70c3094c37fe905

2 years agoAdd management command to reset and update Hiccup statistics
Borjan Tchakaloff [Fri, 16 Mar 2018 07:04:47 +0000 (11:04 +0400)]
Add management command to reset and update Hiccup statistics

Introduce a new management command `stats` that allows to reset and
update the Hiccup statistics easily. Resetting the statistics means
removing all of them and computing them again. Updating the statistics
means resuming counting from the last timestamp. To that end, a new
model `StatsMetadata` is created to hold the last known timestamp the
statistics were computed at.

The `stats` command is meant to be run at regular intervals. For example
with `cron`:

    0 0,6,12,18 * * * python /path/to/hiccup/manage.py stats update

FIXME:
    The counting algorithm drops duplicates based on their timestamp to
    avoid duplicate heartbeats. The true deduplication should happen on
    (`device_id`, `date`) instead. Better yet, the database should be
    cleared of those duplicates altogether.

Issue: HIC-112
Change-Id: I975c896bddbc1308d156f0dbd90d0b66d2d5529b

2 years agoApply the linters to crashreport_stats/models.py
Borjan Tchakaloff [Fri, 3 Aug 2018 15:00:23 +0000 (17:00 +0200)]
Apply the linters to crashreport_stats/models.py

Follow the linters recommendations about the code-style and
add some long-awaited documentation.

Change-Id: I588c579418d1c21227f1e587b74648eb3e49c71b

2 years agoLet pylint validate the import order
Borjan Tchakaloff [Fri, 3 Aug 2018 14:55:15 +0000 (16:55 +0200)]
Let pylint validate the import order

Since the plugin pylint-django is installed, pylint is aware of
Django's recommended import order. Remove flake8's plugin that
was redundant (and following different rules).

Change-Id: I65c63a412a15e30df7c7fa431a9088e1b820d24d

2 years agoSwitch to tox
Borjan Tchakaloff [Thu, 2 Aug 2018 14:26:34 +0000 (16:26 +0200)]
Switch to tox

Use the tox virtual environment testing framework to make our
lives easier. The code style and linter programs really belong
there, in the testing environment.

The tox configuration also defines a linter environement
(`tox -e linters`) that is not useful right now. The state of
the codebase is too bad and we will transition step-by-step.
Instead, we have to make sure the new patchsets are validated
and to that extent the linters should be run manually for now.
A pre-commit hook can be installed to run flake8 automatically.

The two linters included with this patch (flake8 and pylint)
use fixed versions to have a consistent linting experience.
They also have their own requirements file since they can be
used independently (or combined, as with tox).

Also, remove the following dependencies:
  - autopep8 as it is not part of the flow (yet);
  - coverage as it is not part of the flow (yet);
  - mccabe (installed by flake8);
  - pep8 that is the older name of pycodestyle (installed by
    flake8);
  - pyflakes (installed by flake8).

Change-Id: Ib5b6ef2860cb83788adee52b4c4716a5b634bc10

2 years agoExtend crashreport dummy data
Borjan Tchakaloff [Mon, 19 Mar 2018 14:33:24 +0000 (18:33 +0400)]
Extend crashreport dummy data

Add ability to select what type of crashreport data is to be created.

Issue: HIC-144
Change-Id: I544ebebba95afab83829aa5a306e5248270ef784

2 years agoAdd constants for boot reasons
Franz-Xaver Geiger [Mon, 16 Apr 2018 13:00:14 +0000 (15:00 +0200)]
Add constants for boot reasons

Create constants for known boot reason in the Crashreport model.

All occurences of these strings are changed to use the constants, except
raw SQL queries.

Issue: HIC-143
Change-Id: I358a05b0dde408f9a5595c96c37c2dddaeb2b215

2 years agoRestrict the statistics access to staff members
Borjan Tchakaloff [Mon, 9 Apr 2018 14:16:11 +0000 (16:16 +0200)]
Restrict the statistics access to staff members

- Simplify how to decide if an user belongs to the staff;
- Any (Daily)Version stats access requires staff access.

Issue: HIC-142
Change-Id: Iea76ca1c085d2984515250225930c1f39c09742f

2 years agoIntroduce REST endpoints for radio version models
Borjan Tchakaloff [Fri, 23 Feb 2018 13:03:49 +0000 (17:03 +0400)]
Introduce REST endpoints for radio version models

Issue: HIC-141
Change-Id: Id8ca50dda82d2a4854e8987c69ccc847ced4de71