Mercurial Hosting > traffic-intelligence

annotate scripts/merge-features.py @ 971:9897a13772fb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added utils to load video sequence in metadata
author Nicolas Saunier <nicolas.saunier@polymtl.ca>
date Wed, 13 Dec 2017 14:06:20 -0500
parents 75530d8c0090
children 7463c9bc846b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
1 #! /usr/bin/env python
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
2
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
3 import sys, argparse, os.path, sqlite3
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
4 import cvutils, utils, moving, storage
971
9897a13772fb added utils to load video sequence in metadata
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 842
diff changeset
5 from metadata import createDatabase, Site, VideoSequence, CameraView, getSite
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
6 from datetime import datetime, timedelta
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
7
971
9897a13772fb added utils to load video sequence in metadata
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 842
diff changeset
8 timeConverter = utils.TimeConverter()
9897a13772fb added utils to load video sequence in metadata
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 842
diff changeset
9
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
10 parser = argparse.ArgumentParser(description='The program merges feature trajectories recorded from the same site synchronously between start and end time.')
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
11 parser.add_argument('-i', dest = 'metadataFilename', help = 'name of the metadata file', required = True)
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
12 parser.add_argument('-n', dest = 'siteId', help = 'site id or site name', required = True)
971
9897a13772fb added utils to load video sequence in metadata
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 842
diff changeset
13 parser.add_argument('-f', dest = 'startTime', help = 'time to start merging features (format %%Y-%%m-%%d %%H:%%M:%%S, eg 2011-06-22 10:00:39)', type = timeConverter.convert) # if not provided, take common time interval
9897a13772fb added utils to load video sequence in metadata
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 842
diff changeset
14 parser.add_argument('-l', dest = 'endTime', help = 'time to stop merging features (format %%Y-%%m-%%d %%H:%%M:%%S, eg 2011-06-22 10:00:39)', type = timeConverter.convert)
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
15 parser.add_argument('-o', dest = 'outputDBFilename', help = 'name of the output SQLite file', required = True)
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
16
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
17 args = parser.parse_args()
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
18
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
19 session = createDatabase(args.metadataFilename)
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
20
971
9897a13772fb added utils to load video sequence in metadata
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 842
diff changeset
21 site = getSite(session, args.siteId)
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
22 if site is None:
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
23 print('Site {} was not found in {}. Exiting'.format(args.siteId, args.metadataFilename))
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
24 sys.exit()
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
25 else:
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
26 site = site[0]
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
27
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
28 startTime = datetime.strptime(args.startTime, utils.datetimeFormat)
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
29 endTime = datetime.strptime(args.endTime, utils.datetimeFormat)
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
30 processInterval = moving.TimeInterval(startTime, endTime)
836
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
31 cameraViews = session.query(CameraView).filter(CameraView.site == site).filter(CameraView.virtual == False)
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
32 videoSequences = session.query(VideoSequence).filter(VideoSequence.virtual == False).order_by(VideoSequence.startTime.asc()).all() #.order_by(VideoSequence.cameraViewIdx) .filter(VideoSequence.startTime <= startTime)
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
33 videoSequences = [vs for vs in videoSequences if vs.cameraView in cameraViews]
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
34 #timeIntervals = [v.intersection(startTime, endTime) for v in videoSequences]
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
35 #cameraViews = set([v.cameraView for v in videoSequences])
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
36
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
37 videoSequences = {cv: [v for v in videoSequences if v.cameraView == cv] for cv in cameraViews}
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
38 timeIntervals = {}
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
39 for cv in videoSequences:
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
40 timeIntervals[cv] = moving.TimeInterval.unionIntervals([v.getTimeInterval() for v in videoSequences[cv]])
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
41
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
42 # intersection of the time interval (union) for each camera view
828
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
43 commonTimeInterval = timeIntervals.values()[0]
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
44 for inter in timeIntervals.values()[1:]:
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
45 commonTimeInterval = moving.TimeInterval.intersection(commonTimeInterval, inter)
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
46 commonTimeInterval = moving.TimeInterval.intersection(commonTimeInterval, processInterval)
14e4ad7c7420 work on merging data for synchronized views
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents:
diff changeset
47
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
48 if commonTimeInterval.empty():
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
49 print('Empty time interval. Exiting')
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
50 sys.exit()
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
51
829
0ddcc41663f5 renaming
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 828
diff changeset
52 if len(set([cv.cameraType.frameRate for cv in cameraViews])) > 1:
0ddcc41663f5 renaming
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 828
diff changeset
53 print('Different framerates of the cameras ({}) are not handled yet. Exiting'.format([cv.cameraType.frameRate for cv in cameraViews]))
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
54 else:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
55 frameRate = cv.cameraType.frameRate
829
0ddcc41663f5 renaming
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 828
diff changeset
56
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
57 try:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
58 outConnection = sqlite3.connect(args.outputDBFilename)
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
59 outCursor = outConnection.cursor()
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
60 storage.createTrajectoryTable(outCursor, 'positions')
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
61 storage.createTrajectoryTable(outCursor, 'velocities')
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
62 storage.createFeatureCorrespondenceTable(outCursor)
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
63 outConnection.commit()
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
64 except sqlite3.OperationalError as error:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
65 storage.printDBError(error)
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
66 sys.exit()
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
67
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
68 dirname = os.path.split(args.metadataFilename)[0]
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
69 if len(dirname) == 0:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
70 dirname = '.'
829
0ddcc41663f5 renaming
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 828
diff changeset
71
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
72 newTrajectoryId = -1
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
73 # first frame num is commonTimeInterval
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
74 for cv, vs in videoSequences.iteritems():
836
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
75 print cv.idx, cv.description
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
76 for videoSequence in vs:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
77 try:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
78 vsConnection = sqlite3.connect(dirname+os.path.sep+videoSequence.getDatabaseFilename())
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
79 vsCursor = vsConnection.cursor()
837
e01cabca4c55 minor modifications to merge-features
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 836
diff changeset
80 firstFrameNum = utils.deltaFrames(videoSequence.startTime, commonTimeInterval.first, frameRate)
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
81 lastFrameNum = (commonTimeInterval.last-videoSequence.startTime).seconds*frameRate
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
82 # positions table
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
83 vsCursor.execute('SELECT * FROM positions WHERE frame_number BETWEEN {} AND {} ORDER BY trajectory_id'.format(firstFrameNum, lastFrameNum))
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
84 featureIdCorrespondences = {}
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
85 currentTrajectoryId = -1
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
86 for row in vsCursor:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
87 if row[0] != currentTrajectoryId:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
88 currentTrajectoryId = row[0]
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
89 newTrajectoryId += 1
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
90 featureIdCorrespondences[currentTrajectoryId] = newTrajectoryId
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
91 outCursor.execute(storage.insertTrajectoryQuery('positions'), (newTrajectoryId, row[1]-firstFrameNum, row[2], row[3]))
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
92 # velocities table
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
93 for row in vsCursor:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
94 outCursor.execute(storage.insertTrajectoryQuery('velocities'), (featureIdCorrespondences[row[0]], row[1]-firstFrameNum, row[2], row[3]))
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
95 # saving the id correspondences
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
96 for oldId, newId in featureIdCorrespondences.iteritems():
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
97 outCursor.execute("INSERT INTO feature_correspondences (trajectory_id, source_dbname, db_trajectory_id) VALUES ({},\"{}\",{})".format(newId, videoSequence.name, oldId))
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
98 outConnection.commit()
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
99 except sqlite3.OperationalError as error:
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
100 storage.printDBError(error)
836
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
101
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
102 # save the information of the new virtual sequence and camera view in the metadata
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
103 mergedCameraView = CameraView('merged', None, site, cv.cameraType, None, None, virtual = True)
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
104 session.add(mergedCameraView)
7058a40a4bbc updated metadata and code to merge features from different cameras
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 830
diff changeset
105 session.add(VideoSequence('merged', commonTimeInterval.first, commonTimeInterval.last-commonTimeInterval.first, mergedCameraView, virtual = True))
830
2a5856961933 first working version of feature merging (works with feature grouping)
Nicolas Saunier <nicolas.saunier@polymtl.ca>
parents: 829
diff changeset
106 session.commit()