A transform selection for a multi-transform video encoding scheme (AVC possible extension) is presented. This transform selection uses rate distortion optimization (RDO) technique. This method can also be adapted to be used for coding mode-selection and quantization step selection. RDO technique is applied to a selection of transform for each residual sub-block (4x4 PELs) from an available transform collection. The mode selection methodology from  is improved and used for the transform selection. The  mode selection process uses suboptimal RDO process with one dimensional space instead of two-dimensional space, which leads to a PSNR degradation of 1-2dB compared to two-dimensional space RDO. The suggested RDO method uses two stage minimization processes: Initially, Lagrange minimization is applied to each transform. Then, second Lagrange minimization used to select transform/quantization pair. The complexity is reduced by assuming that the RD function of the scalar quantization is a monotonic function, and by use in developed bit-rate predictor. The bit-rate prediction improves the methods that are presented in  and  and achieves bit-rate degradation of 0.31% - 6.8% compared to non prediction scheme vs. a bit-rate degradation of 3.2% - 13.2% by use bitrate predictor which presented in  and . Finally video encoding process which presented achieve up to 30% of improvement. The video encoding uses RDO selection and the bitrate predictor which developed here. The RDO process selects out of three alternative transform which given by .