Mercurial Hosting > traffic-intelligence

annotate scripts/compute-clearmot.py @ 982:51d8406b2489

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
corrected bug when not using a homography and using a mask for CLEAR MOT metrics
author Nicolas Saunier <nicolas.saunier@polymtl.ca>
date Tue, 27 Feb 2018 16:45:46 -0500
parents 196a1fd498ba
children 933670761a57
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
1 #! /usr/bin/env python
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
2
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
3 import sys, argparse
592
985a3021cff2 first match table implementation
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 591
diff changeset
4 from numpy import loadtxt
726
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
5 from numpy.linalg import inv
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
6 import moving, storage, cvutils
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
7
592
985a3021cff2 first match table implementation
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 591
diff changeset
8 # TODO: need to trim objects to same mask ?
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
9
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
10 parser = argparse.ArgumentParser(description='The program computes the CLEAR MOT metrics between ground truth and tracker output (in Polytrack format)', epilog='''CLEAR MOT metrics information:
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
11 Keni, Bernardin, and Stiefelhagen Rainer. "Evaluating multiple object tracking performance: the CLEAR MOT metrics." EURASIP Journal on Image and Video Processing 2008 (2008)
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
12
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
13 Polytrack format:
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
14 JP. Jodoin\'s MSc thesis (in french)
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
15 see examples on http://www.jpjodoin.com/urbantracker/dataset.html''', formatter_class=argparse.RawDescriptionHelpFormatter)
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
16 parser.add_argument('-d', dest = 'trackerDatabaseFilename', help = 'name of the Sqlite database containing the tracker output', required = True)
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
17 parser.add_argument('-g', dest = 'groundTruthDatabaseFilename', help = 'name of the Sqlite database containing the ground truth', required = True)
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
18 parser.add_argument('-o', dest = 'homographyFilename', help = 'name of the filename for the homography (if tracking was done using the homography)')
594
9e39cd95e017 first implementation of CLEAR MOT (needs formal tests)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 593
diff changeset
19 parser.add_argument('-m', dest = 'matchingDistance', help = 'matching distance between tracker and ground truth trajectories', required = True, type = float)
726
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
20 parser.add_argument('--mask', dest = 'maskFilename', help = 'filename of the mask file used to define the where objects were tracked')
594
9e39cd95e017 first implementation of CLEAR MOT (needs formal tests)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 593
diff changeset
21 parser.add_argument('-f', dest = 'firstInstant', help = 'first instant for measurement', required = True, type = int)
9e39cd95e017 first implementation of CLEAR MOT (needs formal tests)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 593
diff changeset
22 parser.add_argument('-l', dest = 'lastInstant', help = 'last instant for measurement', required = True, type = int)
785
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
23 parser.add_argument('--offset', dest = 'nFramesOffsetAnnotations', help = 'number of frames to offset the ground truth annotations', type = int)
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
24 parser.add_argument('--displayOffset', dest = 'nFramesOffsetDisplay', help = 'number of frames to offset annotations and objects for display', type = int)
723
e14e2101a5a9 returns detailed matching information for clear mot
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 636
diff changeset
25 parser.add_argument('--display', dest = 'display', help = 'display the ground truth to object matches (graphically)', action = 'store_true')
823
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
26 parser.add_argument('--undistort', dest = 'undistort', help = 'undistort the video (because features have been extracted that way)', action = 'store_true')
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
27 parser.add_argument('--intrinsic', dest = 'intrinsicCameraMatrixFilename', help = 'name of the intrinsic camera file')
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
28 parser.add_argument('--distortion-coefficients', dest = 'distortionCoefficients', help = 'distortion coefficients', nargs = '*', type = float)
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
29 parser.add_argument('--undistorted-multiplication', dest = 'undistortedImageMultiplication', help = 'undistorted image multiplication', type = float)
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
30
726
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
31 parser.add_argument('-i', dest = 'videoFilename', help = 'name of the video file (for display)')
874
a1e92c48dfb4 correcting error
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 873
diff changeset
32 parser.add_argument('--csv', dest = 'csvOutput', help = 'output comma-separated metrics', action = 'store_true')
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
33 args = parser.parse_args()
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
34
636
3058e00887bc removed all issues because of tests with None, using is instead of == or !=
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 595
diff changeset
35 if args.homographyFilename is not None:
982
51d8406b2489 corrected bug when not using a homography and using a mask for CLEAR MOT metrics
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 956
diff changeset
36 invHomography = inv(loadtxt(args.homographyFilename))
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
37 else:
982
51d8406b2489 corrected bug when not using a homography and using a mask for CLEAR MOT metrics
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 956
diff changeset
38 invHomography = None
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
39
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
40 objects = storage.loadTrajectoriesFromSqlite(args.trackerDatabaseFilename, 'object')
726
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
41
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
42 if args.maskFilename is not None:
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
43 maskObjects = []
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
44 from matplotlib.pyplot import imread
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
45 mask = imread(args.maskFilename)
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
46 if len(mask) > 1:
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
47 mask = mask[:,:,0]
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
48 for obj in objects:
982
51d8406b2489 corrected bug when not using a homography and using a mask for CLEAR MOT metrics
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 956
diff changeset
49 maskObjects += obj.getObjectsInMask(mask, invHomography, 10) # TODO add option to keep object if at least one feature in mask
726
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
50 objects = maskObjects
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
51
768
f8e0a8ea8402 updated the bounding box code (the name so that it is general for ground truth and UT)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 726
diff changeset
52 annotations = storage.loadBBMovingObjectsFromSqlite(args.groundTruthDatabaseFilename)
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
53 for a in annotations:
982
51d8406b2489 corrected bug when not using a homography and using a mask for CLEAR MOT metrics
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 956
diff changeset
54 a.computeCentroidTrajectory(invHomography)
591
aded6c1c2ebd added framework script and function to compute matchings
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
55
785
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
56 if args.nFramesOffsetAnnotations is not None:
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
57 for a in annotations:
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
58 a.shiftTimeInterval(args.nFramesOffsetAnnotations)
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
59
956
196a1fd498ba removing unnecessary complexity of varying number of return elements
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 875
diff changeset
60 motp, mota, mt, mme, fpt, gt, gtMatches, toMatches = moving.computeClearMOT(annotations, objects, args.matchingDistance, args.firstInstant, args.lastInstant, args.display)
723
e14e2101a5a9 returns detailed matching information for clear mot
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 636
diff changeset
61
873
6b474db46b45 added csv printing option
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 823
diff changeset
62 if args.csvOutput:
875
45384b45e35a correcting error
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 874
diff changeset
63 print('{},{},{},{},{}'.format(motp, mota, mt, mme, fpt))
873
6b474db46b45 added csv printing option
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 823
diff changeset
64 else:
6b474db46b45 added csv printing option
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 823
diff changeset
65 print 'MOTP: {}'.format(motp)
6b474db46b45 added csv printing option
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 823
diff changeset
66 print 'MOTA: {}'.format(mota)
6b474db46b45 added csv printing option
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 823
diff changeset
67 print 'Number of missed objects.frames: {}'.format(mt)
6b474db46b45 added csv printing option
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 823
diff changeset
68 print 'Number of mismatches: {}'.format(mme)
6b474db46b45 added csv printing option
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 823
diff changeset
69 print 'Number of false alarms.frames: {}'.format(fpt)
785
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
70
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
71 def shiftMatches(matches, offset):
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
72 shifted = {}
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
73 for k in matches:
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
74 shifted[k] = {t+offset:v for t, v in matches[k].iteritems()}
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
75 return shifted
3aa6102ccc12 addressed issues with ground truth annotations shifted in time
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 768
diff changeset
76
723
e14e2101a5a9 returns detailed matching information for clear mot
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 636
diff changeset
77 if args.display:
823
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
78 if args.undistort and args.intrinsicCameraMatrixFilename is not None:
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
79 intrinsicCameraMatrix = loadtxt(args.intrinsicCameraMatrixFilename)
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
80 else:
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
81 intrinsicCameraMatrix = None
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
82 firstInstant = args.firstInstant
f6790357f53b corrected bug for replay, replaying now from (potentially distorted) video
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 785
diff changeset
83 lastInstant = args.lastInstant
982
51d8406b2489 corrected bug when not using a homography and using a mask for CLEAR MOT metrics
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 956
diff changeset
84 cvutils.displayTrajectories(args.videoFilename, objects, {}, invHomography, firstInstant, lastInstant, annotations = annotations, undistort = args.undistort, intrinsicCameraMatrix = intrinsicCameraMatrix, distortionCoefficients = args.distortionCoefficients, undistortedImageMultiplication = args.undistortedImageMultiplication, gtMatches = gtMatches, toMatches = toMatches)
726
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
85 #print('Ground truth matches')
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
86 #print(gtMatches)
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
87 #print('Object matches')
43ae3a1af290 added functionality to display matchings between ground truth and tracked objects
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 723
diff changeset
88 #rint toMatches