Scenarist reports bad m2v while importing

[Thingy]

I have encoded a m2v stream with tmpgenc and when I import it in Scenarist I get the following error:

Error : Number of fields (40) in GOP(# 114) or between sequence_header_code should be equal or smaller than 36.

Info 0 file(s) accepted, 1 file(s) rejected

I think it's because the GOP header contains bad data that it has 20 interlaced frames (2x20 = 40) while the main header says the GOPs have 18 frames: IBBPBBPBBPBBPBBPBB. The stream is interlaced ntsc.

1. Anyone knows if this is a known bug with tmpgenc or scenarist?

2. Other Authoring programs like DVD-Maestro does not complain over the file. But I really like to use Scenarist. So how do I fix this stream?

3. Rempeg do fix it but it reencodes the stream to a lower bitrate which I rather don't want. Anyone knows where I can get the source for it. If it is available?

4. Tried some header-patchers but none of those fiddle with the field information of the GOPs. so they doesn't actually fix the stream.

Thanks in advance and Hail to setarip, your knowledge is formidable!

[SaSi]

This is not a Scenarist bug. The DVD specifications dictate that a sequence header and a GOP header exists in the stream after N frames, where N differs between PAL and NTSC. I know it must be less than 30, but can't remember the actual values for each system off hand.

The lack of the Sequence header/ GOP header was a bug in Tmpgenc, in old versions. Make sure you are using a recent version first of all.

Then, the easiest way to proceed is to load a predefined template (the DVD PAL or DVD NTSC) template which you can unlock by subsequently loading the unlock.mcf.

The fact that other authoring programs accept some video streams Scenarist rejects is mainly to do with the fact that Scenarist does a very strict conformance test (compared to DVD Maestro or DVD Lab or others). This is irritating (at first) but ensures that the DVD authoried will be proprely playable on desktop players. The lack of Sequence Headers and GOP headers causes problems with fast forward and even more problems with Reverse playback.

[Thingy]

Thanks for the info.

Followup query: Is there any program that can cut out based on GOPs from a mpeg2 file? ie cut out for example GOP #120 or a set like GOP #110 -> #130.
so I could reencode just these frames.

I mean it would make this specific problem go away in a much faster rate than reencoding the whole movie. Or as in the Rempeg2 case I would just reduce a very small part of my file...

[SaSi]

I am not sure exactly what you want to achieve. To avoid re-encoding with DVD compliant settings, you would want to insert Sequence headers and GOP headers every 30 or so pictures.

Theoretically (and practically), it's possible for a program to read an MPEG-2 video stream and insert Sequence headers (and extensions) followed by GOP headers every N pictures and perhaps remove the existing ones if they are not where they should be.

However, this program should be able to count frames and based on the frame rate of the video, stamp each GOP header with the correct timecode/frame, otherwise the GOP headers might cause even further confusion.

I'm afraid I don't know of such a program, but perhaps Restream might correct this for you. Try at shh.dvdboard.de/reastream.html

[Thingy]

I don't think that would be very wise, since GOPs to my knowledge need to start with an I-frame.

What I want is to cut the mpeg stream at particular GOPs in this example
file1: GOP#1 - #119
file2: GOP#120
file3: GOP#121 ->

But all programs I found, only cut based on timestamps.

Well I'm going to spend a few hours writing such a program. It shouldn' t cause me so much anguish...

Thanks for your input and future such also

[setarip]

If you haven't already downloaded and tried "MPEG Restream", as suggested by SaSi, the following information from its included readme file may give you a sense as to whether you can use it to resolve your problem:

" ReStream v0.8.9
==============================
by shh
eMail: shh(at)dvdboard.de
homepage: http://shh.dvdboard.de

Info:
======
With this tool you can change many options of a MPEG2 Elementary Stream
without re-encoding. You can change options like the Sequence Header,
matrices, tff-flags, User Data, GOP-flags, pan-scan-info ...

Intention:
===========
- change/insert/strip/save manual matrices. This is to fix faulty streams
wich lack of the correct matrices. (e.g. from bad demultiplexors or from
a patched CCEv2.50)
- correct top-field-first flags
- change aspect-ratio or framerate
- change the nominal bitrate (needed for some authoring-programs)
- disguise a stream to improve compatibility with some authoring-programs
- remove User Data from the stream
- remove sequence-end-codes from the stream
- add/change/remove Sequence Display Extensions (e.g. for pan-scan info)
- correct/change GOP-info, like the GOP-timecode
- correct some other parts of a stream
- cut a stream at a spectfic sequence header"

[SaSi]

If you are going to write such a program, then here are some further tips.

1. Decide if you are going to work on a demuxed set of audio and video files or a program stream (i.e. m2v + mp2 or .mpg). The program stream contains the video and audio as packets. Obviously, working on a program stream is simpler, as you can split video and audio at the same time. But you must make sure you are not cutting at the end of a video sequence leaving the audio packets that may be just after that segment.

2. If you are going to work with large files (> 2Gb) then make sure you are either working with C++ or something like that, or if you work with VB use the API calls to open a file stream. VB is limited to opening files directly at 2Gb size.

3. If you search for GOPs, don't expect to find GOP Ids like you describe them. GOPS are not numbered. They are time stamped. The time stamps are a combination of time and frames. For example, the first GOP has a time/frame stamp of 0:00:00 + 0 frames. In a 25fps video, if the second GOP starts at frame 18, then the second GOP's time/frame stamp will read 0:00:00 + 18 frames. If the third GOP starts at frame 28 then its time/frame stamp will read 0:00:01 + 3 frames.

4. To parse an MPEG stream is of course possible but you must first study the MPEG-2 video and system specifications very closely. I am writing an MPEG parser program with the final target of being able to compare the quality between an unencoded AVI file and the encoded MPEG video - to compare encoders and settings. I have already spent 2 weeks of reading and programming and the groundwork for spliting a video file is already implemented, but not for a system stream as well. Expect to spend at least 2 weeks (unless you have already read and understood the 13818-1 document).

5. Finaly, try one of the good MPEG editors around. I would suggest your try Mainconcept's EVE or Honestech video editor. They can both edit MPEG files and of course they all cut at I-frame boundary.

6. The trick and problem at cuting an MPEG file is the start frame. Being at an I-Frame boundary is not enough. If the video stream was encoded with Open GOPs, then one of the P or B frames following the I frame may reference a previous frame that belongs to the previous GOP. Even if you encode with Open GOPs, such a reference is not very likely to happen in every GOP, but it really may happen, so when you cut, you must make sure you check backward references closely before you decide that any I-frame position is a suitable candidate for cutting.

[Thingy]

Yepp, I have been starting to noticing such issues, when I am halfway through my task
I have come to the conclusion that the smartest thing might be to cut the GOPs that are to long in half. I will first try to just insert sequence and GOP headers before the middle P-frame and thus fooling scenarist to accept the stream (who knows it's picky, but it might not be smart). If that wont work I will try to convert the P-frame to an I-frame and also inserting SEQ and GOP headers, this oughto be enough since I am not touching B frames. Hopefully B-frames aren't decoded based on if the previous trailing reference-frame is an P or I-frame...

I hope your code looks better than mine, I'm using c so it's not the most readable around but atleast I insert one or two /* comments */

Good Luck to me and I hope you have a succesful comparing scheme worked out for you.

Note: how do you calculate difference?

and the below statement you can disregard from since Why don't apply to quality...

[SaSi]

If I remember correctly, according to the MPEG-2 video specs, the first frame following a GOP header must be an I frame. If I am correct, then you can't do what you are trying to.

In any case, for valid MPEG-2 video stream, you need to have
Sequence Header
Sequence Extension
Group header
Picture header
Picture coding extension
slice 1
slice 2
...
slice n
Picture Header
Picture Coding Extension
slice 1
...
slice n
...
Sequence Header
Sequence Extension
Group header
...


About comparing frames, the idea is to make several calculations towards evaluating the average weighted difference. I am not yet even close starting this, I first need to decode the macroblocks and compose the decoded frame, which I have only half done.

My ambition is to generate a graph showing bitrate vs weighted quality and calculate an average weighted quality rating.

[Thingy]

I have put the programming-project on ice til I solve another issue I experience:

I changed the authoring software to DVDMaestro, very nice program even if it's not as comprehensive as Scenarist...

I patch the m2v-file with DVDPatcher so the headers read 352x480, as needed, works fine.
I import it to Maestro works fine, in fact everythin is fine, except when I try to play it on my standalone. My HOYO 8050D experience the Pioneer issue, with only showing 2/3 of the image.
Ok, I have a 16:9 tv so I could just turn it to 16:9 and watch. BUT I loose pixels... I have walked all over the bloody net to find info on this sde-patching that works for Pioneer drives.

Steps I take: (omitting sound, cause that works)

1. extract the m2v
2. Patch headers with DVDPatcher to 352x480.
3. Author
4. Repatch Headers to 480x480.

I even tried ReStream to insert the sde value. I tried both 720 and 480 for horizontal value. no difference.
I tried patching the headers with DVDPatcher and set it to 16:9. And something happened. Instead of stretching the movie sideways it clipped it. If anyone knows the steps I need to be taking in what order and what values that works for the Pioneer drive I'm sure I can make all this go away...

[SaSi]

DVDMaestro is as complete as Scenarist is. I don't think there are DVD features that DVDMaestro can't handle. (For example, all the recent James Bond editions (Region 2) are done with this.

Now, on the patching and clipping problem.

Firsrt of all, what are the real dimensions of the encoded pictures?

What do you mean with sde patching? Do you mean Sequence Display Extension patching? If yes, then I can't understand how patching the SDE can affect the way the video is displayed. For MP@ML video, all the required information is retrieved from the Sequence Header (frame dimensions, frame rate and aspect ratio).

[Thingy]

My encoded video is origainally from a svcd. so it's 480x480.

The sde-patching refers to that some DVD-players (Pioneer) uses these values for clipping and resizing the final video before displaying.

The main procedure for creating a svcd-dvd without reencoding the video (which I would prefer to not do)
1. Patch m2v header so the authoring software doesn't complain over a bad rezolution, ie 480x480
(Have noted that DVDMaestro crashes if the m2v is patched to 704x480, 352x480 works though)

2. Author your svcd-dvd.

3. RePatch the VOB- headers to 480x480, so the dvd-decoder decodes the video correctly. (This works)

The problem is that DVD-decoders reading the sde (which they shouldn't). They use these values to do funny things.

I first thought that my hoyo 8050d would be capable of getting this right, appearantly it doesn't read the sde cause there is no change what so ever if I change the sde-resolution.

What I have concluded is that DVDMaestro doesn't do anything with the SDE, it is leaving all m2v headers alone.