# HG changeset patch # User mdd # Date 1511892121 -3600 # Node ID 82361ad7b3fe5e9daccc28bc6d5c018f2bf26aa6 # Parent b398ae3883283ef9c86d73b19ee25a65d6a52e21 some changes, also implemented ffmpeg progress info and added force overwrite mode diff -r b398ae388328 -r 82361ad7b3fe dupecheck.py --- a/dupecheck.py Sat Nov 25 16:51:08 2017 +0100 +++ b/dupecheck.py Tue Nov 28 19:02:01 2017 +0100 @@ -24,6 +24,9 @@ elif filename.endswith(".mkv"): title = filename[:-4].lower() FILELIST.append([title, filename, root]) + elif filename.endswith(".mp4"): + title = filename[:-4].lower() + FILELIST.append([title, filename, root]) print("%i files found, running duplicate testing loop" % len(FILELIST)) listlen = len(FILELIST) diff -r b398ae388328 -r 82361ad7b3fe ts2mkv.py --- a/ts2mkv.py Sat Nov 25 16:51:08 2017 +0100 +++ b/ts2mkv.py Tue Nov 28 19:02:01 2017 +0100 @@ -9,8 +9,9 @@ """ import subprocess +import pexpect from eit import readeit, eitinfo -import os, shlex +import os, shlex, sys, time def filter_lines(data, search): """ @@ -27,6 +28,7 @@ def run_command(command): """ run command as blocking subprocess, returns exit code + if total_frames > 0 parse ffmpeg status line and insert ETA at line start before output """ process = subprocess.Popen(shlex.split(command), \ stdout=subprocess.PIPE) @@ -39,11 +41,52 @@ rc = process.poll() return rc +def run_ffmpeg_watch(command, frames_total = 0): + """ + run command as blocking subprocess, returns exit code + if total_frames > 0 parse ffmpeg status line and insert ETA at line start before output + """ + thread = pexpect.spawn(command) + cpl = thread.compile_pattern_list([ + pexpect.EOF, + "frame= *(\d+)", + '(.+)' + ]) + percent = 0 + eta = 0 + time_start = time.time() - 0.1 # start in the past + while True: + i = thread.expect_list(cpl, timeout=None) + if i == 0: # EOF + print "the sub process exited" + break + elif i == 1: + try: + frame_number = int(thread.match.group(1)) + if frames_total > 0: + percent = frame_number * 100.00 / frames_total + eta = frame_number / (time.time() - time_start) + # eta is frames per second so far + eta = (frames_total - frame_number) / eta / 60 + sys.stdout.write("\rFrame %i of %i, %.1f%% done, ETA %.0f minutes, " % ( + frame_number, frames_total, percent, eta + )) + except: + sys.stdout.write(thread.match.group(0)) + sys.stdout.flush() + thread.close + elif i == 2: + unknown_line = thread.match.group(0) + sys.stdout.write(unknown_line) + sys.stdout.flush() + pass + def ffmpeg_filename(filename): """ Escape filename path contents for ffmpeg shell command """ - fn = "\\'".join(p for p in filename.split("'")) + #fn = "\\'".join(p for p in filename.split("'")) + fn = filename.replace("'", "\\'") fn = fn.replace(" ", "\\ ") return fn @@ -58,6 +101,9 @@ self.command = None self.filename = None self.outfilebase = None + self.fps = 0 + self.frames_total = 0 + self.overwrite = False self.scaleto_720p = scaleto_720p self.rename = rename @@ -131,6 +177,7 @@ """ lines = filter_lines(self.msg_ffmpeg, "[Parsed_cropdetect").split("\n") option = None + failcount = 0 for line in lines: tmp = line[line.find(" crop="):].strip() #print "DEBUG: " + tmp @@ -138,8 +185,11 @@ option = tmp else: if option != tmp: - self.msg_prepare += "WARNING: cropdetect inconsistent over scan time, disabling autocrop\n" - return None + failcount += 1 + if failcount > 12: + print "!!! Crop detect is inconsistent" + self.msg_prepare += "WARNING: cropdetect >50% inconsistent over scan time, disabling autocrop\n" + return None self.msg_prepare += "Crop detected: %s\n" % option return option @@ -169,7 +219,7 @@ audio = filter_lines(audioall, "(eng):") aidx = self.get_stream_index( filter_lines(audio, "ac3")) - if aidx != "": + if aidx != "" and filter_lines(audio, "ac3").find(" 0 channels ") < 1: # append english audio too! print "Selecting english ac3 stream." audiomap.append(aidx) @@ -189,7 +239,7 @@ outfn = self.outfilebase + ".mkv" # double-check: pull the kill switch and exit if outfile exists already! # we do not want to overwrite files in accident (caused by automatic file naming) - if len(glob.glob(outfn)) > 0: + if not self.overwrite and len(glob.glob(outfn)) > 0: print "Output file exists: %s" % outfn print "NOT overwriting it!" return None @@ -197,7 +247,7 @@ cmd = [ "ffmpeg", "-hide_banner", - "-ss 00:05:00", "-t 1", # search to 5 minutes, analyze 1 second + "-ss 00:05:00", "-t 2", # search to 5 minutes, analyze 2 seconds "-i %s" % fn, "-vf \"cropdetect=24:2:0\"", # detect black bar crop on top and bottom "-f null", "-" # no output file @@ -217,13 +267,39 @@ print "No video stream found" return None + # get total duration and fps from input stream + # Input #0, mpegts, from '/srv/storage0/DREAMBOX/Action/Transporter/20101201 0630 - Sky Action HD - Transporter 3.ts': + # Duration: 01:39:59.88, start: 93674.825111, bitrate: 9365 kb/s + # Stream #0:1[0x1ff]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 50 tbr, 90k tbn, 50 tbc + self.frames_total = filter_lines(self.msg_ffmpeg, "Duration:").strip()[10:] + self.frames_total = self.frames_total[0:self.frames_total.find(",")].strip() + print "Input duration: %s" % self.frames_total + try: + self.frames_total = int(self.frames_total[0:2]) * 3600 + \ + int(self.frames_total[3:5]) * 60 + int(self.frames_total[6:8]) + except ValueError: + self.frames_total = 0 - # TODO: copy ALL subtitle streams if present! + tmp = filter_lines(info, "Video:").split(",") + for fps in tmp: + if fps.strip().endswith('fps'): + try: + self.fps = float(fps.strip().split(' ')[0]) + except ValueError: + self.fps = 0 + break + self.frames_total = round(self.frames_total * self.fps, 0) + print "Input framerate: %f fps" % self.fps + print "Total frames of input file: %i" % (self.frames_total) + + # copy ALL subtitle streams if present! # Stream #0:0[0x20](deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006), 492x250 submap = [] for tmp in filter_lines(info, "Subtitle: dvb_teletext").split("\n"): if self.get_stream_index(tmp): submap.append(self.get_stream_index(tmp)) + # Subtitles disabled, that doesnt work as expected, dreambox crashes on copied subtitle stream + submap = [] # select audio streams audiomap = self.__get_audiomap(info) @@ -248,6 +324,10 @@ .replace('\ ', ' ')\ .replace("\'", "'")\ + "\"" + # no ETA calculation possible since we have only the length of first file + # TODO: we COULD estimate by multiplying with factor generated by input file sizes + print "NO ETA POSSIBLE" + self.frames_total = 0 idx = 0 for tmp in inputs: @@ -260,6 +340,9 @@ "-i %s" % fn, ] + if self.overwrite: + cmd.append("-y") + for tmp in submap: self.msg_prepare += "Subtitle Stream selected: Stream #%s\n" % tmp cmd.append("-map %s" % tmp) @@ -297,6 +380,8 @@ self.msg_prepare = "" self.msg_eit = "" self.msg_ffmpeg = "" + self.fps = 0 + self.frames_total = 0 self.filename = filename self.outfilebase = os.path.splitext(filename)[0] @@ -321,7 +406,8 @@ for cmd in self.command: print "Executing ffmpeg:\n%s\n" % cmd - return run_command(cmd) + #return run_command(cmd, self.total_frames) + return run_ffmpeg_watch(cmd, frames_total=self.frames_total) @@ -340,10 +426,13 @@ help='rename file basename to name and genre from EIT file if present') parser.add_argument('input', metavar='input', nargs='+', \ help='one or more files, glob style syntax') + parser.add_argument('-f', action='store_true', default=False, \ + help='force overwrite of existing file') args = parser.parse_args() processor = ts2mkv(crf=args.crf, tune=args.tune, scaleto_720p=(not args.ns), \ rename=args.rename) + processor.overwrite = args.f for srcstr in args.input: src = glob.glob(srcstr)