Mercurial Hosting > traffic-intelligence

annotate scripts/learn-motion-patterns.py @ 927:c030f735c594

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added assignment of trajectories to prototypes and cleanup of insert queries
author Nicolas Saunier <nicolas.saunier@polymtl.ca>
date Tue, 11 Jul 2017 17:56:23 -0400
parents acb5379c5fd7
children 584b9405e494
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
1 #! /usr/bin/env python
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
2
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
3 import sys, argparse
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
4
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
5 #import matplotlib.pyplot as plt
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
6 import numpy as np
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
7
921
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
8 import ml, utils, storage, moving
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
9
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
10 parser = argparse.ArgumentParser(description='The program learns prototypes for the motion patterns') #, epilog = ''
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
11 #parser.add_argument('--cfg', dest = 'configFilename', help = 'name of the configuration file')
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
12 parser.add_argument('-d', dest = 'databaseFilename', help = 'name of the Sqlite database file', required = True)
921
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
13 parser.add_argument('-o', dest = 'outputPrototypeDatabaseFilename', help = 'name of the Sqlite database file to save prototypes')
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
14 parser.add_argument('-i', dest = 'inputPrototypeDatabaseFilename', help = 'name of the Sqlite database file for prototypes to start the algorithm with')
878
8e8ec4ece66e minor + bug corrected in motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 870
diff changeset
15 parser.add_argument('-t', dest = 'trajectoryType', help = 'type of trajectories to learn from', choices = ['objectfeatures', 'feature', 'object'], default = 'objectfeatures')
920
499154254f37 improved prototype loading
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 919
diff changeset
16 parser.add_argument('--max-nobjectfeatures', dest = 'maxNObjectFeatures', help = 'maximum number of features per object to load', type = int, default = 1)
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
17 parser.add_argument('-n', dest = 'nTrajectories', help = 'number of the object or feature trajectories to load', type = int, default = None)
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
18 parser.add_argument('-e', dest = 'epsilon', help = 'distance for the similarity of trajectory points', type = float, required = True)
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
19 parser.add_argument('--metric', dest = 'metric', help = 'metric for the similarity of trajectory points', default = 'cityblock') # default is manhattan distance
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
20 parser.add_argument('-s', dest = 'minSimilarity', help = 'minimum similarity to put a trajectory in a cluster', type = float, required = True)
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
21 parser.add_argument('-c', dest = 'minClusterSize', help = 'minimum cluster size', type = int, default = None)
908
b297525b2cbf added options to the prototype cluster algorithm, work in progress
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 907
diff changeset
22 parser.add_argument('--optimize', dest = 'optimizeCentroid', help = 'recompute centroid at each assignment', action = 'store_true')
843
5dc7a507353e updated to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 835
diff changeset
23 parser.add_argument('--random', dest = 'randomInitialization', help = 'random initialization of clustering algorithm', action = 'store_true')
907
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
24 parser.add_argument('--subsample', dest = 'positionSubsamplingRate', help = 'rate of position subsampling (1 every n positions)', type = int)
843
5dc7a507353e updated to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 835
diff changeset
25 parser.add_argument('--display', dest = 'display', help = 'display trajectories', action = 'store_true')
5dc7a507353e updated to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 835
diff changeset
26 parser.add_argument('--save-similarities', dest = 'saveSimilarities', help = 'save computed similarities (in addition to prototypes)', action = 'store_true')
927
c030f735c594 added assignment of trajectories to prototypes and cleanup of insert queries
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 922
diff changeset
27 parser.add_argument('--save-matches', dest = 'saveMatches', help = 'saves the assignments of the objects (not for features) to the prototypes', action = 'store_true')
c030f735c594 added assignment of trajectories to prototypes and cleanup of insert queries
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 922
diff changeset
28 #parser.add_argument('--assign', dest = 'assign', help = 'saves the assignments of the objects (not for features) to the prototypes', action = 'store_true') # default is train, but one could want only to assign the objects to the loaded prototypes without learning
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
29
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
30 args = parser.parse_args()
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
31
907
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
32 # use cases
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
33 # 1. learn proto from one file, save in same or another (with traj)
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
34 # 2. load proto, load objects, update proto, save proto
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
35 # 3. assign objects from one db to proto
920
499154254f37 improved prototype loading
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 919
diff changeset
36 # 4. load objects from several files, save in another -> see metadata: site with view and times
917
89cc05867c4c reorg and work in progress
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 910
diff changeset
37 # 5. keep prototypes, with positions/velocities, in separate db (keep link to original data through filename, type and index)
907
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
38
921
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
39 # TODO add possibility to clutesr with velocities
907
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
40 # TODO add possibility to start with saved prototypes so that one can incrementally learn from several databases
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
41 # save the objects that match the prototypes
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
42 # write an assignment function for objects
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
43
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
44 trajectoryType = args.trajectoryType
878
8e8ec4ece66e minor + bug corrected in motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 870
diff changeset
45 prototypeType = args.trajectoryType
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
46 if args.trajectoryType == 'objectfeatures':
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
47 trajectoryType = 'object'
878
8e8ec4ece66e minor + bug corrected in motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 870
diff changeset
48 prototypeType = 'feature'
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
49
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
50 if args.trajectoryType == 'objectfeatures':
910
b58a1061a717 loading is faster for longest object features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 908
diff changeset
51 objectFeatureNumbers = storage.loadObjectFeatureFrameNumbers(args.databaseFilename, objectNumbers = args.nTrajectories)
b58a1061a717 loading is faster for longest object features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 908
diff changeset
52 featureNumbers = []
b58a1061a717 loading is faster for longest object features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 908
diff changeset
53 for numbers in objectFeatureNumbers.values():
b58a1061a717 loading is faster for longest object features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 908
diff changeset
54 featureNumbers += numbers[:min(len(numbers), args.maxNObjectFeatures)]
b58a1061a717 loading is faster for longest object features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 908
diff changeset
55 objects = storage.loadTrajectoriesFromSqlite(args.databaseFilename, 'feature', objectNumbers = featureNumbers, timeStep = args.positionSubsamplingRate)
b58a1061a717 loading is faster for longest object features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 908
diff changeset
56 else:
b58a1061a717 loading is faster for longest object features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 908
diff changeset
57 objects = storage.loadTrajectoriesFromSqlite(args.databaseFilename, trajectoryType, withFeatures = (args.trajectoryType == 'objectfeatures'), objectNumbers = args.nTrajectories, timeStep = args.positionSubsamplingRate)
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
58
921
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
59 if args.inputPrototypeDatabaseFilename is not None:
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
60 prototypeIndices, dbFilenames, trajectoryTypes, nMatchings, prototypes = storage.loadPrototypesFromSqlite(args.inputPrototypeDatabaseFilename, True)
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
61
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
62 trajectories = [o.getPositions().asArray().T for o in objects]
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
63
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
64 lcss = utils.LCSS(metric = args.metric, epsilon = args.epsilon)
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
65 nTrajectories = len(trajectories)
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
66
735
0e875a7f5759 modified prototypeCluster algorithm to enforce similarity when re-assigning and to compute only the necessary similarities
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 734
diff changeset
67 similarities = -np.ones((nTrajectories, nTrajectories))
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
68
908
b297525b2cbf added options to the prototype cluster algorithm, work in progress
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 907
diff changeset
69 prototypeIndices, labels = ml.prototypeCluster(trajectories, similarities, args.minSimilarity, lambda x,y : lcss.computeNormalized(x, y), args.minClusterSize, args.optimizeCentroid, args.randomInitialization, True, None) # this line can be called again without reinitializing similarities
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
70
844
5a68779d7777 added capability to save prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 843
diff changeset
71 clusterSizes = ml.computeClusterSizes(labels, prototypeIndices, -1)
5a68779d7777 added capability to save prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 843
diff changeset
72 print(clusterSizes)
5a68779d7777 added capability to save prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 843
diff changeset
73
922
acb5379c5fd7 corrected
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 921
diff changeset
74 prototypes = [moving.Prototype(args.databaseFilename, objects[i].getNum(), prototypeType, clusterSizes[i]) for i in prototypeIndices]
921
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
75 if args.outputPrototypeDatabaseFilename is None:
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
76 outputPrototypeDatabaseFilename = args.databaseFilename
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
77 else:
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
78 outputPrototypeDatabaseFilename = args.outputPrototypeDatabaseFilename
630934595871 work in progress with prototype class
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 920
diff changeset
79 storage.savePrototypesToSqlite(outputPrototypeDatabaseFilename, prototypes)
818
181bcb6dad3a added option to learn motion patterns and show to display results
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 735
diff changeset
80
843
5dc7a507353e updated to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 835
diff changeset
81 if args.saveSimilarities:
844
5a68779d7777 added capability to save prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 843
diff changeset
82 np.savetxt(utils.removeExtension(args.databaseFilename)+'-prototype-similarities.txt.gz', similarities, '%.4f')
843
5dc7a507353e updated to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 835
diff changeset
83
927
c030f735c594 added assignment of trajectories to prototypes and cleanup of insert queries
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 922
diff changeset
84 labelsToProtoIndices = {protoId: i for i, protoId in enumerate(prototypeIndices)}
c030f735c594 added assignment of trajectories to prototypes and cleanup of insert queries
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 922
diff changeset
85 if args.saveMatches: # or args.assign
c030f735c594 added assignment of trajectories to prototypes and cleanup of insert queries
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 922
diff changeset
86 # save in the db that contained originally the data
c030f735c594 added assignment of trajectories to prototypes and cleanup of insert queries
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 922
diff changeset
87 # retirer les assignations anterieures?
c030f735c594 added assignment of trajectories to prototypes and cleanup of insert queries
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 922
diff changeset
88 storage.savePrototypeAssignmentsToSqlite(args.databaseFilename, objects, [labelsToProtoIndices[l] for l in labels], prototypes)
907
9fd7b18f75b4 re arranged motion pattern learning
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 878
diff changeset
89
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
90 if args.display:
844
5a68779d7777 added capability to save prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 843
diff changeset
91 from matplotlib.pyplot import figure, show, axis
735
0e875a7f5759 modified prototypeCluster algorithm to enforce similarity when re-assigning and to compute only the necessary similarities
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 734
diff changeset
92 figure()
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
93 for i,o in enumerate(objects):
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
94 if i not in prototypeIndices:
735
0e875a7f5759 modified prototypeCluster algorithm to enforce similarity when re-assigning and to compute only the necessary similarities
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 734
diff changeset
95 if labels[i] < 0:
0e875a7f5759 modified prototypeCluster algorithm to enforce similarity when re-assigning and to compute only the necessary similarities
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 734
diff changeset
96 o.plot('kx')
0e875a7f5759 modified prototypeCluster algorithm to enforce similarity when re-assigning and to compute only the necessary similarities
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 734
diff changeset
97 else:
0e875a7f5759 modified prototypeCluster algorithm to enforce similarity when re-assigning and to compute only the necessary similarities
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 734
diff changeset
98 o.plot(utils.colors[labels[i]])
734
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
99 for i in prototypeIndices:
1d4dcb5c8708 first example script to learn prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
100 objects[i].plot(utils.colors[i]+'o')
844
5a68779d7777 added capability to save prototypes
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 843
diff changeset
101 axis('equal')
818
181bcb6dad3a added option to learn motion patterns and show to display results
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 735
diff changeset
102 show()